{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提到分类模型评估相信大家应该都不会觉得陌生（不陌生你点进来干嘛[捂脸]），本文就分类模型评估的基本原理进行讲解，并手把手、肩并肩地带您实现各种评估函数。完整实现代码请参考本人的p...哦不是...github：  \n",
    "- [model_selection.py](https://github.com/tushushu/imylu/blob/master/imylu/utils/model_selection.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 分类模型\n",
    "分类问题其实在生活中处处可见，比如我们在大街上看妹纸，会把妹纸的外貌分为好看和非常好看(求生欲。。。)；再比如我们刷微博，会把微博推送给我们的内容分为喜欢的和不喜欢的。上述问题就是典型的分类问题，确切的说是二分类问题，而能够解决这些二分类问题的数学模型就被称为二分类模型。用数学的方式表达就是，给定自变量X，代入到我们的分类模型F，会输出因变量y，y的取值为0或1，其中0代表负样本（好看的妹纸、不喜欢的微博），1代表正样本（非常好看的妹纸、喜欢的微博）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 准确率\n",
    "如何评估一个分类模型计算出来的结果是否准确呢？最简单的方式就是用准确率(Accuracy)来评价。比如我们手机上有5条微博，我们对这5条微博的态度分别是喜欢、喜欢、不喜欢、不喜欢、喜欢，用数组表示就是y = [1, 1, 0, 0, 1]。我们将微博的文字内容，图片内容等特征X代入到分类模型F中，得到分类结果是y_hat = [1, 1, 0, 0, 0]。显然5个样本中我们正确分类了4个，准确率 = 4 / 5 * 100% = 80%。用Python实现计算准确率的函数如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_acc(y, y_hat):\n",
    "    return sum(yi == yi_hat for yi, yi_hat in zip(y, y_hat)) / len(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 准确率的陷阱\n",
    "如果用准确率就足以评估分类模型的优劣，那么我也就不会专门写一篇文章了[微笑]。假设大街上我们遇到好看的妹纸的概率是99%，遇到非常好看的妹纸的概率是1%。那么我们用模型F(X) = 0，即不分青红皂白，一律简单粗暴地预测所有妹纸都是好看的（负样本），模型就可以达到99%的准确率，这显然是非常荒谬的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 混淆矩阵\n",
    "如何解决准确率的陷阱呢，接下来轮到混淆矩阵矩阵出场了。混淆矩阵，顾名思义，就是可以让你混淆各种模型评价指标的矩阵。矩阵的形状是2 x 2，其中，\n",
    "- 矩阵的左上角表示，预测值为1，实际值为1(True Positive，简称TP)；\n",
    "- 右上角表示预测值为1，实际值为0(False Positive，简称FP)；\n",
    "- 左下角表示预测值为0，实际值为1(False Negative，简称FN)；\n",
    "- 右下角表示预测值为0，实际值为0(True Negative，简称TN)；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. TPR\n",
    "$TPR = TP / NumberOfPositive$  \n",
    "TPR也被称作召回率，即正例被我们准确预测的比例。我们再回头看看准确率的陷阱，简单粗暴地预测所有妹纸都是好看的（负样本），模型就可以达到99%的准确率。但是其TPR是0，即非常好看的妹纸都没有被模型识别出来。用全宇宙最简单的编程语言Python实现TPR的计算函数如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tpr(y, y_hat):\n",
    "    true_positive = sum(yi and yi_hat for yi, yi_hat in zip(y, y_hat))\n",
    "    actual_positive = sum(y)\n",
    "    return true_positive / actual_positive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Precision\n",
    "$Precision = TP / NumberOfPredictedPositive$  \n",
    "Precision也被称作精确率，即我们预测的正例中有多少个是准确的，可以从另一个角度来评估模型的预测能力。用全宇宙最简单的编程语言Python实现Precision的计算函数如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_precision(y, y_hat):\n",
    "    true_positive = sum(yi and yi_hat for yi, yi_hat in zip(y, y_hat))\n",
    "    predicted_positive = sum(y_hat)\n",
    "    return true_positive / predicted_positive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. TNR\n",
    "$TNR = TN / NumberOfNegative$  \n",
    "TNR也称特异度，即我们预测的负例中有多少个是准确的。除此之外，特异度更多地被用于ROC曲线的绘制。用全宇宙最简单的编程语言Python实现TNR的计算函数如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tnr(y, y_hat):\n",
    "    true_negative = sum(1 - (yi or yi_hat) for yi, yi_hat in zip(y, y_hat))\n",
    "    actual_negative = len(y) - sum(y)\n",
    "    return true_negative / actual_negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. 分类模型的阈值\n",
    "为了简化问题，前面我们讨论分类模型的输出都是0和1的离散变量。事实上分类模型一般会输出一个介于0和1之间的数字x，比如0.75。我们需要设定一个阈值k，默认是0.5也可以根据实际情况调整。如果x >= k，那么预测结果就是1，否则预测结果就是0。由于0.75 >= 0.5，所以预测结果是1。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. ROC\n",
    "由于TPR = TP / Number of Positive，所以当阈值为0时，所有的样本都会被预测为Positive，所以TPR等于1，同理这时的1 - TNR为1，当阈值为1时TPR等于0，1- TNR等于0。如果我们让阈值从1逐渐降低到0，会得到很多对TPR和1 - TNR，将这些值当做数据点，以TPR为y轴，1 - TNR为x轴绘制出一条曲线，这条曲线就是ROC曲线。用全宇宙最简单的编程语言Python实现ROC数据点的计算函数如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_roc(y, y_hat_prob):\n",
    "    thresholds = sorted(set(y_hat_prob), reverse=True)\n",
    "    ret = [[0, 0]]\n",
    "    for threshold in thresholds:\n",
    "        y_hat = [int(yi_hat_prob >= threshold) for yi_hat_prob in y_hat_prob]\n",
    "        ret.append([get_tpr(y, y_hat), 1 - get_tnr(y, y_hat)])\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. AUC\n",
    "ROC曲线下的面积被称为AUC，可以评估模型的性能。用全宇宙最简单的编程语言Python实现AUC的计算函数如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_auc(y, y_hat_prob):\n",
    "    roc = iter(get_roc(y, y_hat_prob))\n",
    "    tpr_pre, fpr_pre = next(roc)\n",
    "    auc = 0\n",
    "    for tpr, fpr in roc:\n",
    "        auc += (tpr + tpr_pre) * (fpr - fpr_pre) / 2\n",
    "        tpr_pre = tpr\n",
    "        fpr_pre = fpr\n",
    "    return auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1 AUC为0.5，模型没有预测能力"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们生成1000个实际值，其中500个值为1,500个值为0，顺序被随机打乱。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from numpy.random import rand, seed, shuffle, normal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(100)\n",
    "y = np.array([0, 1] * 500)\n",
    "shuffle(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们随机生成1000个实际值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(20)\n",
    "y_pred = rand(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。不难看出这条ROC曲线的AUC值约等于0.5，而预测值都是我们随机生成的，也就印证了AUC为0.5时模型没有预测能力的说法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.479.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9df07f0>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFc5JREFUeJzt3XusXeV55/Hvg0NiNcVJjIPGxRifqo7EKVM16SkGIQVQ0sgmKlajNIEEtZlS3But0kwrpSQiCZEml6YTpRo6yZmUpIlqyEUqdlo3KE0JriLsYBSaxKbMeDAxB9BgTApqO3S4PPPH3hsvtvc5e/mctS9rre9HsrQvr/d5l8/xz6+f9V4iM5EkNctpk+6AJKl6hrskNZDhLkkNZLhLUgMZ7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1EAvmdQXXrduXW7atGlSX16Saumee+55PDNfPazdxMJ906ZNHDhwYFJfXpJqKSJ+WKadZRlJaiDDXZIayHCXpAaaWM19kGeeeYaFhQWefvrpSXdlxVavXs2GDRs4/fTTJ90VSS00VeG+sLDAGWecwaZNm4iISXdn2TKT48ePs7CwwMzMzKS7I6mFhpZlIuLmiHgsIn6wyPsREX8aEYcj4nsR8brldubpp5/mzDPPrHWwA0QEZ555ZiP+ByKpnsrU3D8PbF3i/W3A5u6vHcB/X0mH6h7sPU25Dkn1NDTcM3Mv8MQSTbYDX8iOfcArI2J9VR2UpKbYuf8ob//MXXzoawdH/rWqqLmfDTxUeL7Qfe3R/oYRsYPO6J6NGzdW8KUlabrt3H+UXfc+DMD+I51x8uxPrBn5160i3AfVHwaeup2Z88A8wNzc3NSfzJ2ZZCanneaMUUmnphfqvUDfMrOWLTNr2f6zZ/OOLaMf3FYR7gvAOYXnG4BHKvjciXjwwQfZtm0bl112GXfddRfvfve7+cQnPkFm8uY3v5mPfexjAHz961/n+uuv57nnnmPdunV885vfnHDPJU2DQaE+rkAvqiLcdwPXRcStwBbgycw8qSRzqj70tYMceuSpFXeuaPYn1vCBX/zpoe3uv/9+Pve5z/H+97+fCy+8kHvuuYdXvepVvOlNb+K2227j4osv5tprr2Xv3r3MzMzwxBNL3ZKQ1BY79x/l+r/6PjC5UO8ZGu4RcQtwKbAuIhaADwCnA2Tmp4E9wOXAYeDfgP80qs6Oy7nnnsuFF17Irl27uPTSS3n1qzsbsL3zne9k7969rFq1ite//vUvzGFfu3btJLsraQoUg/2//NJ/nFio9wwN98y8asj7CfxOZT3qKjPCHpWXv/zlQKfmPkhmOtVREnByGWYagh3cW2ZJW7Zs4c477+Txxx/nueee45ZbbuGSSy7hoosu4s477+TIkSMAlmWkFupNa7z+r77P/iNPsGVm7dQEO0zZ9gPTZv369XzkIx/hsssuIzO5/PLL2b59OwDz8/O85S1v4fnnn+ess87iG9/4xoR7K2mcdt37MIcefWritfXFxGKlh1Gbm5vL/sM67rvvPs4777yJ9GcUmnY9kk6UYQ49+hSz69fwpd+4aKxfPyLuycy5Ye0cuUtSSYNmw0wrw12SSpi22TDDTN0N1UmViarWlOuQVL9ghykL99WrV3P8+PHaB2NvP/fVq1dPuiuSKtDbG6YuwQ5TVpbZsGEDCwsLHDt2bNJdWbHeSUyS6qm44VdvVkxdgh2mLNxPP/10Ty6SNHbFIO8p7g0zu37NVN88HWSqwl2SJqE4tbFnWuevl2W4S2q1nfuPvrDCdNxz1kdpqm6oStK49coxdSu7DOPIXVIrFVea1u1maRmGu6TWqdNK0+Uy3CW1Th3nrZ8qw11SazS9FFNkuEtqhTaUYooMd0mNVVycNG0nJY2a4S6psYqLk+q+KOlUGe6SGm0SB2pMAxcxSWqk3srTtnLkLqkxBtXYm37jdDGGu6RG6J8N07Yaez/DXVLt1fGkpFEz3CXVUpunOZZhuEuqHUswwxnukmqnDXvDrJRTISXVUtP3hlkpR+6Sptqg8037j8TTyRy5S5pqvS0Eiup4YPW4lRq5R8RW4FPAKuCzmfnRvvc3An8BvLLb5r2ZuafivkpqieJovTdKb+MWAisxdOQeEauAm4BtwCxwVUTM9jV7P/DlzHwtcCXwZ1V3VFJ7FEfrjtKXp8zI/QLgcGY+ABARtwLbgUOFNgn0CmCvAB6pspOSmq2/ru5ofeXK1NzPBh4qPF/ovlb0QeDqiFgA9gC/W0nvJLVCf13d0frKlRm5x4DXsu/5VcDnM/NPIuIi4IsRcX5mPv+iD4rYAewA2LjRKUySTnCkXq0y4b4AnFN4voGTyy7XAFsBMvOuiFgNrAMeKzbKzHlgHmBubq7/HwhJLTLopqmqUybc7wY2R8QM8DCdG6bv6GtzFHgD8PmIOA9YDRyrsqOS6m/QfjBbZtZahhmBoeGemc9GxHXA7XSmOd6cmQcj4kbgQGbuBv4z8D8i4vfplGzelZmOzCW9wP1gxqvUPPfunPU9fa/dUHh8CLi42q5Jqjt3bpwctx+QVLleqBdLL47Ux8twl1Sp/vKLgT4Zhrukyngi0vRw4zBJlXGf9elhuEuqlPusTwfLMpJWrHcD1cVI08Nwl7Rsg2bFuBhpOhjukk7JYqtMnRUzXQx3SaekWH4x1KeX4S7plLmD4/RztowkNZAjd0lDuT1v/ThylzSUZ5rWjyN3SaVYZ68Xw13SSRY7sFr1YbhLesGgRUlgKaaODHdJgFv1No3hLrVc/2jdHR2bwXCXWmqxfWEM9mYw3KUWcV+Y9jDcpRZxX5j2MNylhuqfzggnpjQ6X735XKEqNVBv5kuv9NLjlMb2cOQuNYyHVAscuUuN4yHVAsNdapSd+4+y/8gTHlItyzJSE/TPWbeuLsNdqjEXImkxhrtUQ4a6hjHcpRrqLUYy1LUYw12qiUFH3bkYSYspNVsmIrZGxP0RcTgi3rtIm7dFxKGIOBgRO6vtptRu/YuSXIykYYaO3CNiFXAT8AvAAnB3ROzOzEOFNpuBPwIuzswfRcRZo+qw1EbOXdepKlOWuQA4nJkPAETErcB24FChzbXATZn5I4DMfKzqjkpt01+Gce66TkWZsszZwEOF5wvd14peA7wmIr4dEfsiYuugD4qIHRFxICIOHDt2bHk9llrAMoxWqszIPQa8lgM+ZzNwKbAB+IeIOD8z//lFvylzHpgHmJub6/8MSV2WYbRSZUbuC8A5hecbgEcGtNmVmc9k5hHgfjphL+kUuYWAqlAm3O8GNkfETES8FLgS2N3X5jbgMoCIWEenTPNAlR2V2qI3arcMo5UYGu6Z+SxwHXA7cB/w5cw8GBE3RsQV3Wa3A8cj4hBwB/CHmXl8VJ2WmspRu6pSahFTZu4B9vS9dkPhcQLv6f6StEyO2lUVt/yVpoyjdlXBcJekBnJvGWkK9BYs9faMkVbKcJcmpLgCtX/rXmmlDHdpzAbtxe7Wvaqa4S6NmXuxaxwMd2mMivPY3Ytdo2S4S2PgAdYaN8NdGrHeDo/gWacaH8NdGpH+0bo7PGqcDHepQktNbzTYNU6Gu1SR/vKLoa5JMtylChSD3fKLpoF7y0grZLBrGhnu0gp5JJ6mkeEurYCHa2haGe7SCni4hqaV4S4tk6N2TTPDXVqG4k1UR+2aRk6FlE6Bq05VF4a7VJJ7xKhODHepBOeyq26suUtDGOyqI8NdWoLBrrqyLCMN4I1T1Z3hLnW5Xa+axHCXcLteNY/hrlaz/KKmMtzVKsXSC1h+UXMZ7mqVXfc+zKFHn2J2/RrAUFdzlQr3iNgKfApYBXw2Mz+6SLu3Al8Bfj4zD1TWS2kFiqP1XrB/6TcumnCvpNEaOs89IlYBNwHbgFngqoiYHdDuDOD3gP1Vd1Jart6N0l75ZXb9Gjf6UiuUGblfABzOzAcAIuJWYDtwqK/dh4GPA39QaQ+lZfBGqdquTLifDTxUeL4AbCk2iIjXAudk5l9HhOGuiekPdWvqaqsy4R4DXssX3ow4Dfgk8K6hHxSxA9gBsHGjf9lULXdtlE4oE+4LwDmF5xuARwrPzwDOB74VEQD/AdgdEVf031TNzHlgHmBubi6RKuIeMNKLldk47G5gc0TMRMRLgSuB3b03M/PJzFyXmZsycxOwDzgp2KVRMdilkw0N98x8FrgOuB24D/hyZh6MiBsj4opRd1BaisEuDVZqnntm7gH29L12wyJtL115t6ThDHZpca5QVa0M2rnRYJdOZrhrKvXvAdNTnOLojBhpcYa7pk7/lMYiA10qx3DX1OmN2C23SMvnGaqaSltm1hrs0goY7poqO/cffaGuLmn5LMto4gbNgHHnRmllDHdNxGKHUXvDVKqG4a6xWOp4OwNdqp7hrrHweDtpvAx3jY3H20njY7hrpHrlmOKoXdLoGe6qzKAtA/pPRJI0Hoa7KjNohG5tXZoMw10rUhyt94Lduro0eYa7lmXQQdSz69dYepGmhOGu0hZbeGTZRZo+hruGGjRKN9Sl6Wa4a0n9e6sb6FI9GO5alGeUSvXllr8ayGCX6s1w10CehiTVm2UZvUhxuwBPQ5Lqy5G7XqS4ytQ561J9OXIXcPIGX64ylerNcNfA6Y6S6s1wb6lBq029eSo1h+HeQv0jdRcnSc1juDfcUnusO1KXmstwb6DFNvjqcaQuNZ/h3jCWXCRByXCPiK3Ap4BVwGcz86N9778H+HXgWeAY8GuZ+cOK+6oSXFkqCUosYoqIVcBNwDZgFrgqImb7mn0XmMvMnwG+Cny86o5qaTv3H+Xtn7nLlaWSgHIj9wuAw5n5AEBE3ApsBw71GmTmHYX2+4Crq+ykBlvq8AxJ7VYm3M8GHio8XwC2LNH+GuBvB70RETuAHQAbNzqyXC4Pz5A0TJlwjwGv5cCGEVcDc8Alg97PzHlgHmBubm7gZ2hpHp4hqYwy4b4AnFN4vgF4pL9RRLwReB9wSWb+ezXdU0//aN0bppKWUibc7wY2R8QM8DBwJfCOYoOIeC3wGWBrZj5WeS/1om14Ha1LGmZouGfmsxFxHXA7namQN2fmwYi4ETiQmbuBPwZ+HPhKRAAczcwrRtjvVtm5/yj7jzzBlpm17tYoqZRS89wzcw+wp++1GwqP31hxv8TJpRhnwUgqyxWqEzRo35ei/umNlmIklWW4T8CgqYyDGOqSlstwH6NBoW54SxoFw71iS5VaDHVJ42K4V6x4Dmk/Q13SuBjuI+AB05ImbeiukJKk+jHcJamBDHdJaiDDvUK9bQIkadK8obpCgw7McJsASZNmuC+TB2ZImmaG+ylY6lg7A13SNDHcT0FxgZKhLmmaGe4luae6pDpxtkwJxXNLvVkqqQ4cuS9iUH3dc0sl1YXhPkBxpO4sGEl1ZLgX9E9vdKQuqa4M967+0bojdUl11vpwd7QuqYlaHe6O1iU1VevC3VkwktqgdeHuKlNJbdDocB90WHUv2F1lKqnJGr1CtTdKL5pdv8ZVppIar5Ej996I3VG6pLZqTLgvtR2vJLVNY8LdG6WSdEJjwh2wBCNJXaXCPSK2Ap8CVgGfzcyP9r3/MuALwM8Bx4G3Z+aD1Xb1ZMVSTG/ULkkqMVsmIlYBNwHbgFngqoiY7Wt2DfCjzPwp4JPAx6ru6CDF2TDOgpGkE8qM3C8ADmfmAwARcSuwHThUaLMd+GD38VeB/xYRkZlZYV9fxJORJGlxZcL9bOChwvMFYMtibTLz2Yh4EjgTeLyKThZ96GsHOfTIUy/MiHG0LkknKxPuMeC1/hF5mTZExA5gB8DGjSubyeKMGElaXJlwXwDOKTzfADyySJuFiHgJ8Argif4Pysx5YB5gbm5uWSWbD/ziTy/nt0lSq5TZfuBuYHNEzETES4Ergd19bXYDv9p9/Fbg70dZb5ckLW3oyL1bQ78OuJ3OVMibM/NgRNwIHMjM3cCfA1+MiMN0RuxXjrLTkqSllZrnnpl7gD19r91QePw08MvVdk2StFyN3hVSktrKcJekBjLcJamBDHdJaiDDXZIaKCY1HT0ijgE/XOZvX8cItjaYcl5zO3jN7bCSaz43M189rNHEwn0lIuJAZs5Nuh/j5DW3g9fcDuO4ZssyktRAhrskNVBdw31+0h2YAK+5Hbzmdhj5Ndey5i5JWlpdR+6SpCVMdbhHxNaIuD8iDkfEewe8/7KI+FL3/f0RsWn8vaxWiWt+T0QciojvRcQ3I+LcSfSzSsOuudDurRGREVH7mRVlrjki3tb9Xh+MiJ3j7mPVSvxsb4yIOyLiu92f78sn0c+qRMTNEfFYRPxgkfcjIv60++fxvYh4XaUdyMyp/EVne+H/Dfwk8FLgH4HZvja/DXy6+/hK4EuT7vcYrvky4Me6j3+rDdfcbXcGsBfYB8xNut9j+D5vBr4LvKr7/KxJ93sM1zwP/Fb38Szw4KT7vcJrfj3wOuAHi7x/OfC3dE6yuxDYX+XXn+aR+wsHc2fm/wN6B3MXbQf+ovv4q8AbImLQkX91MfSaM/OOzPy37tN9dE7GqrMy32eADwMfB54eZ+dGpMw1XwvclJk/AsjMx8bcx6qVueYE1nQfv4KTT3yrlczcy4AT6Qq2A1/Ijn3AKyNifVVff5rDfdDB3P2nYb/oYG6gdzB3XZW55qJr6PzLX2dDrzkiXguck5l/Pc6OjVCZ7/NrgNdExLcjYl9EbB1b70ajzDV/ELg6IhbonB/xu+Pp2sSc6t/3U1LqsI4Jqexg7hopfT0RcTUwB1wy0h6N3pLXHBGnAZ8E3jWuDo1Bme/zS+iUZi6l87+zf4iI8zPzn0fct1Epc81XAZ/PzD+JiIvonO52fmY+P/ruTcRI82uaR+6ncjA3Sx3MXSNlrpmIeCPwPuCKzPz3MfVtVIZd8xnA+cC3IuJBOrXJ3TW/qVr2Z3tXZj6TmUeA++mEfV2VueZrgC8DZOZdwGo6e7A0Vam/78s1zeHexoO5h15zt0TxGTrBXvc6LAy55sx8MjPXZeamzNxE5z7DFZl5YDLdrUSZn+3b6Nw8JyLW0SnTPDDWXlarzDUfBd4AEBHn0Qn3Y2Pt5XjtBn6lO2vmQuDJzHy0sk+f9B3lIXebLwf+J5277O/rvnYjnb/c0PnmfwU4DHwH+MlJ93kM1/x3wP8B7u3+2j3pPo/6mvvafouaz5Yp+X0O4L8Ch4DvA1dOus9juOZZ4Nt0ZtLcC7xp0n1e4fXeAjwKPENnlH4N8JvAbxa+xzd1/zy+X/XPtStUJamBprksI0laJsNdkhrIcJekBjLcJamBDHdJaiDDXa0WEb8XEfdFxF9Oui9SlZwKqVaLiH8CtmVnFeiwtqsy87kxdEtaMUfuaq2I+DSdLWh3R8STEfHFiPj7iPhfEXFtt82l3T3Gd9JZaCLVgiN3tVp3v5o54Drgl+jsXfNyOnupb6Gz7P9vgPPLjO6laeHIXTphV2b+38x8HLiDzh7kAN8x2FU3hrt0Qv9/Y3vP/3XcHZFWynCXTtgeEasj4kw6+6jfPeH+SMtmuEsnfIdOfX0f8OHMrPUxb2o3b6hKQER8EPiXzPzEpPsiVcGRuyQ1kCN3SWogR+6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRAhrskNdD/B0SJDj4LZr8qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.2 AUC值为1，模型的预测能力最强"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们让预测值直接等于实际值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.array(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。不难看出这条ROC曲线的AUC值1，也就印证了AUC值为1，模型的预测能力最强的说法。当然，在实际应用时这种情况是几乎不可能出现的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 1.000.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9e840f0>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAEQVJREFUeJzt3W2MpWV9x/Hvz11wUwo+sEOCLDBruiasWON2AktIeAjW7G4imxprAY21IVC16AtNI40GKb6w1bamprS6sWolFURfwNauUqvINsZFhoAIS7fdAsqIKcNqMa1defr3xTnAdJjduWfmzJyda76fZJJz3/d17vO/9pz57TXXuR9SVUiS2vKiYRcgSRo8w12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUoNXDeuG1a9fW6OjosF5ekpalO++887GqGpmt3dDCfXR0lPHx8WG9vCQtS0l+2KWd0zKS1CDDXZIaZLhLUoMMd0lqkOEuSQ2aNdyTfDbJo0nuPcT2JPlkkv1J7kmyafBlSpLmosvI/fPAlsNs3wps6P9cDvzNwsuSJC3ErMe5V9XuJKOHabId+EL17te3J8lLk5xYVT8ZUI3/zxdv/xE33/3jxdi1JC2Jja84jg+/8dWL+hqDmHM/CXh4yvJEf90LJLk8yXiS8cnJyXm92M13/5i9P/n5vJ4rSSvFIM5QzQzrZrzrdlXtAHYAjI2NzfvO3BtPPI4v/f5Z8326JDVvECP3CeDkKcvrgEcGsF9J0jwNItx3Am/vHzWzGXh8sebbJUndzDotk+R64DxgbZIJ4MPAUQBV9SlgF7AN2A/8Avi9xSpWktRNl6NlLp5lewF/MLCKJEkL5hmqktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGdwj3JliT7kuxPcuUM209JcmuSu5Lck2Tb4EuVJHU1a7gnWQVcC2wFNgIXJ9k4rdmHgBur6nXARcBfD7pQSVJ3XUbuZwD7q+qBqnoCuAHYPq1NAcf1H78EeGRwJUqS5mp1hzYnAQ9PWZ4AzpzW5mrgn5K8BzgGeP1AqpMkzUuXkXtmWFfTli8GPl9V64BtwHVJXrDvJJcnGU8yPjk5OfdqJUmddAn3CeDkKcvreOG0y6XAjQBV9V1gDbB2+o6qakdVjVXV2MjIyPwqliTNqku43wFsSLI+ydH0vjDdOa3Nj4ALAJKcRi/cHZpL0pDMGu5V9RRwBXALcD+9o2LuS3JNkgv7zd4PXJbk+8D1wDuqavrUjSRpiXT5QpWq2gXsmrbuqimP9wJnD7Y0SdJ8eYaqJDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkN6hTuSbYk2Zdkf5IrD9HmLUn2JrkvyRcHW6YkaS5Wz9YgySrgWuA3gQngjiQ7q2rvlDYbgD8Czq6qnyU5YbEKliTNrsvI/Qxgf1U9UFVPADcA26e1uQy4tqp+BlBVjw62TEnSXHQJ95OAh6csT/TXTfUq4FVJvpNkT5ItM+0oyeVJxpOMT05Ozq9iSdKsuoR7ZlhX05ZXAxuA84CLgc8keekLnlS1o6rGqmpsZGRkrrVKkjrqEu4TwMlTltcBj8zQ5uaqerKqHgT20Qt7SdIQdAn3O4ANSdYnORq4CNg5rc1NwPkASdbSm6Z5YJCFSpK6mzXcq+op4ArgFuB+4Maqui/JNUku7De7BTiQZC9wK/CHVXVgsYqWJB3erIdCAlTVLmDXtHVXTXlcwPv6P5KkIfMMVUlqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSgTuGeZEuSfUn2J7nyMO3enKSSjA2uREnSXM0a7klWAdcCW4GNwMVJNs7Q7ljgvcDtgy5SkjQ3XUbuZwD7q+qBqnoCuAHYPkO7jwAfAw4OsD5J0jx0CfeTgIenLE/01z0nyeuAk6vqqwOsTZI0T13CPTOsq+c2Ji8CPgG8f9YdJZcnGU8yPjk52b1KSdKcdAn3CeDkKcvrgEemLB8LnA58O8lDwGZg50xfqlbVjqoaq6qxkZGR+VctSTqsLuF+B7AhyfokRwMXATuf3VhVj1fV2qoarapRYA9wYVWNL0rFkqRZzRruVfUUcAVwC3A/cGNV3ZfkmiQXLnaBkqS5W92lUVXtAnZNW3fVIdqet/CyJEkL4RmqktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1qFO4J9mSZF+S/UmunGH7+5LsTXJPkm8mOXXwpUqSupo13JOsAq4FtgIbgYuTbJzW7C5grKp+HfgK8LFBFypJ6q7LyP0MYH9VPVBVTwA3ANunNqiqW6vqF/3FPcC6wZYpSZqLLuF+EvDwlOWJ/rpDuRT42kwbklyeZDzJ+OTkZPcqJUlz0iXcM8O6mrFh8jZgDPj4TNurakdVjVXV2MjISPcqJUlzsrpDmwng5CnL64BHpjdK8nrgg8C5VfXLwZQnSZqPLiP3O4ANSdYnORq4CNg5tUGS1wGfBi6sqkcHX6YkaS5mDfeqegq4ArgFuB+4saruS3JNkgv7zT4O/Crw5SR3J9l5iN1JkpZAl2kZqmoXsGvauqumPH79gOuSJC2AZ6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNWj1sAuQpEF58sknmZiY4ODBg8MuZcHWrFnDunXrOOqoo+b1fMNdUjMmJiY49thjGR0dJcmwy5m3quLAgQNMTEywfv36ee3DaRlJzTh48CDHH3/8sg52gCQcf/zxC/oLxHCX1JTlHuzPWmg/DHdJapDhLkmLpKp45plnhvLahrskDdBDDz3Eaaedxrvf/W42bdrEddddx2te8xpOP/10PvCBDzzX7utf/zqbNm3ita99LRdccMHA6/BoGUlN+uN/uI+9j/x8oPvc+Irj+PAbXz1ru3379vG5z32OD33oQ2zevJk777yTl73sZbzhDW/gpptu4uyzz+ayyy5j9+7drF+/np/+9KcDrRMMd0kauFNPPZXNmzdz8803c9555zEyMgLAW9/6Vnbv3s2qVas455xznjvM8eUvf/nAa+gU7km2AH8JrAI+U1V/Mm37i4EvAL8BHAB+p6oeGmypktRdlxH2YjnmmGOA3pz7TKpq0Y/qmXXOPckq4FpgK7ARuDjJxmnNLgV+VlW/BnwC+NNBFypJy82ZZ57JbbfdxmOPPcbTTz/N9ddfz7nnnstZZ53FbbfdxoMPPggwtGmZM4D9VfUAQJIbgO3A3ilttgNX9x9/BfirJKlD/bclSSvAiSeeyEc/+lHOP/98qopt27axfft2AHbs2MGb3vQmnnnmGU444QS+8Y1vDPS1u4T7ScDDU5YngDMP1aaqnkryOHA88NggipSk5WJ0dJR77733ueVLLrmESy655AXttm7dytatWxetji7hPtPE0PQReZc2JLkcuBzglFNO6fDSL7TxFcfN63mStJJ0CfcJ4OQpy+uARw7RZiLJauAlwAsmkapqB7ADYGxsbF5TNsP8kkSSlosuJzHdAWxIsj7J0cBFwM5pbXYCv9t//GbgW863S9LwzDpy78+hXwHcQu9QyM9W1X1JrgHGq2on8LfAdUn20xuxX7SYRUvSoSzFYYZLYaHj407HuVfVLmDXtHVXTXl8EPjtBVUiSQu0Zs0aDhw4sOwv+/vs9dzXrFkz7314hqqkZqxbt46JiQkmJyeHXcqCPXsnpvky3CU146ijjpr3nYta41UhJalBhrskNchwl6QGZViHoyeZBH44z6evZeVd2sA+rwz2eWVYSJ9PraqR2RoNLdwXIsl4VY0Nu46lZJ9XBvu8MixFn52WkaQGGe6S1KDlGu47hl3AENjnlcE+rwyL3udlOecuSTq85TpylyQdxhEd7km2JNmXZH+SK2fY/uIkX+pvvz3J6NJXOVgd+vy+JHuT3JPkm0lOHUadgzRbn6e0e3OSSrLsj6zo0uckb+m/1/cl+eJS1zhoHT7bpyS5Ncld/c/3tmHUOShJPpvk0ST3HmJ7knyy/+9xT5JNAy2gqo7IH3qXF/4P4JXA0cD3gY3T2rwb+FT/8UXAl4Zd9xL0+XzgV/qP37US+txvdyywG9gDjA277iV4nzcAdwEv6y+fMOy6l6DPO4B39R9vBB4adt0L7PM5wCbg3kNs3wZ8jd6d7DYDtw/y9Y/kkftzN+auqieAZ2/MPdV24O/6j78CXJDlfJ3PDn2uqlur6hf9xT307oy1nHV5nwE+AnwMOLiUxS2SLn2+DLi2qn4GUFWPLnGNg9alzwU8ex/Nl/DCO74tK1W1mxnuSDfFduAL1bMHeGmSEwf1+kdyuM90Y+6TDtWmqp4Cnr0x93LVpc9TXUrvf/7lbNY+J3kdcHJVfXUpC1tEXd7nVwGvSvKdJHuSbFmy6hZHlz5fDbwtyQS9+0e8Z2lKG5q5/r7PyZF8yd+B3Zh7GencnyRvA8aAcxe1osV32D4neRHwCeAdS1XQEujyPq+mNzVzHr2/zv4lyelV9V+LXNti6dLni4HPV9WfJzmL3t3dTq+qZxa/vKFY1Pw6kkfuc7kxN4e7Mfcy0qXPJHk98EHgwqr65RLVtlhm6/OxwOnAt5M8RG9ucucy/1K162f75qp6sqoeBPbRC/vlqkufLwVuBKiq7wJr6F2DpVWdft/n60gO95V4Y+5Z+9yfovg0vWBf7vOwMEufq+rxqlpbVaNVNUrve4YLq2p8OOUORJfP9k30vjwnyVp60zQPLGmVg9Wlzz8CLgBIchq9cF/+t1Q6tJ3A2/tHzWwGHq+qnwxs78P+RnmWb5u3Af9G71v2D/bXXUPvlxt6b/6Xgf3A94BXDrvmJejzPwP/Cdzd/9k57JoXu8/T2n6bZX60TMf3OcBfAHuBHwAXDbvmJejzRuA79I6kuRt4w7BrXmB/rwd+AjxJb5R+KfBO4J1T3uNr+/8ePxj059ozVCWpQUfytIwkaZ4Md0lqkOEuSQ0y3CWpQYa7JDXIcNeKluS9Se5P8vfDrkUaJA+F1IqW5F+BrdU7C3S2tquq6uklKEtaMEfuWrGSfIreJWh3Jnk8yXVJvpXk35Nc1m9zXv8a41+kd6KJtCw4cteK1r9ezRhwBfBb9K5dcwy9a6mfSe+0/38ETu8yupeOFI7cpefdXFX/W1WPAbfSuwY5wPcMdi03hrv0vOl/xj67/D9LXYi0UIa79LztSdYkOZ7eddTvGHI90rwZ7tLzvkdvfn0P8JGqWta3edPK5heqEpDkauC/q+rPhl2LNAiO3CWpQY7cJalBjtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg/4POzvMpRcbgkoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.3 对于正例和负例有着对等预测能力的ROC曲线"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "阈值为0.5时，我们让预测值有70%的可能性是正确预测。比如yi = 1，那么预测值有70%的可能性是[0.5, 1]之间的随机数。如果yi = 0，那么预测值有70%的可能性是[0, 0.5]之间的随机数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(15)\n",
    "f = lambda x: rand() / 2 + 0.5 if x else rand() / 2\n",
    "y_pred = np.array([f(yi) if rand() > 0.3 else f(1 - yi) for yi in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。可以看出ROC曲线的形状是对称的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.706.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9ed2c18>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFUBJREFUeJzt3X+s3fV93/HnGwdmjeEEsJEcG+NbzZFw6ZRER9goUgrKDxmiYiXKWtNEayuE0y7QZK0mMRIlGZGaJU1XrRpq4rVJ2mj8aqTi29YNSymRqwg7vlZYEsy8eUDhBjQMyUBb5gbCe3+cc+zD4dx7vvf6e873e77f50OydL7nfH3v+8u1X3z8/n6+n09kJpKkZjmn6gIkSeUz3CWpgQx3SWogw12SGshwl6QGMtwlqYEMd0lqIMNdkhrIcJekBnpdVd94/fr1uXXr1qq+vSTNpKNHjz6XmRvGnVdZuG/dupWFhYWqvr0kzaSI+Psi59mWkaQGMtwlqYEMd0lqoMp67qO89NJLLC4ucurUqapLOWtr165l8+bNnHvuuVWXIqmFahXui4uLXHDBBWzdupWIqLqcVctMnn/+eRYXF5mbm6u6HEktNLYtExFfiohnI+L7S3weEfEHEXEiIr4bEW9dbTGnTp3i4osvnulgB4gILr744kb8C0TSbCrSc/8KsGuZz68FtvV+7QX+8GwKmvVg72vKdUiaTWPbMpl5MCK2LnPKbuBPs7tf36GIeENEbMzMZ0qqUZJm2p2Hn2T/wz84fbz9jev45C/87ES/Zxk9903AUwPHi733XhPuEbGX7uieLVu2lPCtJameBgP98OM/BGDH3EVT+/5lhPuo/sPIXbczcx+wD6DT6dR+Z+7MJDM55xxnjEoq7s7DT3Lbn38P6Ab6jrmL2P3mTfzyjukNassI90Xg0oHjzcDTJXzdSjzxxBNce+21XHPNNTz00EN89KMf5fOf/zyZyXve8x4++9nPAvD1r3+d2267jZ/+9KesX7+eBx54oOLKJVVhuOUCZ0bqv/Pen5tqoA8qI9zngZsj4m5gB/BCGf32f/sXj3Ds6RfPurhBRftcx48f58tf/jIf//jH2blzJ0ePHuXCCy/k3e9+N/fddx9ve9vbuOmmmzh48CBzc3P88Ic/LLVOSbNheITeV8VIfdjYcI+Iu4CrgfURsQh8EjgXIDO/ABwArgNOAD8Gfm1SxU7LZZddxs6dO9m/fz9XX301GzZ0F2D7wAc+wMGDB1mzZg1vf/vbT89hv+ii6fXRJE3XqJF5Xx1G6EspMlvmhjGfJ/Dh0irqmfSd5OWcf/75QLfnPkpmOtVRarh+qC93M7QOI/Sl1OoJ1brZsWMHH/nIR3juuee48MILueuuu7jlllu46qqr+PCHP8zjjz9+ui3j6F1qhlGhXtcAX47hvoyNGzfymc98hmuuuYbM5LrrrmP37t0A7Nu3j/e973288sorXHLJJXzjG9+ouFpJq7XUtMVZDPW+WKr1MGmdTieHN+t49NFHufzyyyupZxKadj1Skyw3D73OoR4RRzOzM+48R+6SWqcO89AnzXCX1DjLzXCBes9yKUvtwr0pM1GqandJbbfU3PNBTRypD6tVuK9du5bnn39+5pf97a/nvnbt2qpLkRpveJTehlF5EbUK982bN7O4uMjJkyerLuWs9XdiklSOpVotwzdD2zAqL6JW4X7uuee6c5GkkfY//AOOPfMi2zeue9X7hvlotQp3Se017iZoP9jv+dBVU6xqdhnukiqxVK98qZug2zeuY/ebN02ltiYw3CVN3HLL4tornwzDXdJE1XlZ3CYz3CWtyrgeeZ9TE6thuEtasSIPCvU5Qq+G4S5pWXXdRk7LM9wlnVbkxmf/taPxejPcpZYpsm2cQT77DHepwYqOxPsM8uYw3KUGWm7/TwO8HQx3qWGGZ7IY5O1kuEszqEjf3Jks7Wa4SzNgJeuwOFoXGO5SrS3VOzfANY7hLlWs6NREw1wrYbhLFRgMdFssmgTDXZqSpQLdANckGO7ShI3qmxvomjTDXZqApUbpBrqmxXCXSmLbRXVSKNwjYhfwH4A1wB9l5r8b+nwL8CfAG3rn3JqZB0quVaol2y6qo7HhHhFrgDuAdwGLwJGImM/MYwOnfRy4NzP/MCK2AweArROoV6rccg8UGeiqiyIj9yuBE5n5GEBE3A3sBgbDPYF1vdevB54us0ipDnygSLOkSLhvAp4aOF4Edgyd8yngv0TELcD5wDtLqU6qgVGhbpir7oqEe4x4L4eObwC+kpm/FxFXAV+NiCsy85VXfaGIvcBegC1b/Iuh+nOFRc2qIuG+CFw6cLyZ17ZdbgR2AWTmQxGxFlgPPDt4UmbuA/YBdDqd4f9BSJVyr1A1SZFwPwJsi4g54AfAHuCXh855EngH8JWIuBxYC5wss1CpTO4VqqYbG+6Z+XJE3AzcT3ea45cy85GIuB1YyMx54LeB/xQR/4puy+ZXM9ORuWrBIFcbRVUZ3Ol0cmFhoZLvreYrsjCXQa5ZFBFHM7Mz7jyfUNVMWm6ZXPAJUclw10xZbuPnQQa62s5w10xwrrm0Moa7asuVFaXVM9xVC+NmtBjq0soY7pqKldwA7TPQpdUz3DURy62cOIpBLpXLcFcpxoW54S1Nl+Gusca1VMAwl+rGcNeyhldFXIphLtWL4a4lDQa7qyJKs8Vw12sMPzBksEuzx3DXaT4FKjWH4a7T9j/8A44986KhLjWA4a5X2b5xHfd86Kqqy5B0ls6pugDVw52HnzzdjpE0+xy5t9xwn333mzdVXJGkMhjuLTY8h90+u9QchntLOYddajZ77i3VX07AYJeayXBvsR1zFxnsUkMZ7i3kzBip+Qz3Fuq3ZJwZIzWXN1RbpD/tsf8Uqi0ZqbkM95YYNe1RUnMZ7i3h7BipXey5t4itGKk9HLk33GCfffvGdVWXI2lKHLk33GCw22eX2sORewu4jK/UPo7cJamBCoV7ROyKiOMRcSIibl3inF+MiGMR8UhE3FlumVqpOw8/yS998SGOPfNi1aVIqsDYtkxErAHuAN4FLAJHImI+M48NnLMN+DfA2zLzRxFxyaQKVjH22qV2K9JzvxI4kZmPAUTE3cBu4NjAOTcBd2TmjwAy89myC1Vx/bVjdsxdZK9daqkibZlNwFMDx4u99wa9CXhTRHwrIg5FxK5RXygi9kbEQkQsnDx5cnUVa1mDT6I6Ypfaq0i4x4j3cuj4dcA24GrgBuCPIuINr/lNmfsys5OZnQ0bNqy0VhXgk6iSoFi4LwKXDhxvBp4ecc7+zHwpMx8HjtMNe1XAJ1ElFQn3I8C2iJiLiPOAPcD80Dn3AdcARMR6um2ax8osVOO5TrukvrHhnpkvAzcD9wOPAvdm5iMRcXtEXN877X7g+Yg4BjwI/OvMfH5SRWs012mX1BeZw+3z6eh0OrmwsFDJ926a4fVjnCEjNVdEHM3MzrjzXH5gxrlOu6RRDPcZ5+wYSaO4tswMG3xYyWCXNMhwn2HeQJW0FMN9xjlqlzSKPfcZ5O5KksYx3GeMs2MkFWG4zxhnx0gqwp77DLLPLmkcw12SGshwl6QGsuc+I5whI2klHLnPCPdElbQSjtxniCs+SirKkfsMcBMOSStluM8A15CRtFKGe8258qOk1TDca2xwqQFH7ZJWwnCvqcFgd6kBSStluNeUa8hIOhuGe43ZZ5e0Woa7JDWQ4S5JDeQTqjXjGjKSyuDIvWZcQ0ZSGRy518TwiN01ZCSdDUfuNeGIXVKZHLnXiCN2SWVx5C5JDWS4S1IDFQr3iNgVEccj4kRE3LrMee+PiIyITnklSpJWamy4R8Qa4A7gWmA7cENEbB9x3gXAbwKHyy5SkrQyRUbuVwInMvOxzPwJcDewe8R5nwY+B5wqsT5J0ioUCfdNwFMDx4u9906LiLcAl2bmX5ZYW2u4jZ6kshUJ9xjxXp7+MOIc4PeB3x77hSL2RsRCRCycPHmyeJUN5zZ6kspWJNwXgUsHjjcDTw8cXwBcAXwzIp4AdgLzo26qZua+zOxkZmfDhg2rr7qBXN5XUpmKhPsRYFtEzEXEecAeYL7/YWa+kJnrM3NrZm4FDgHXZ+bCRCqWJI01Ntwz82XgZuB+4FHg3sx8JCJuj4jrJ12gJGnlCi0/kJkHgAND731iiXOvPvuy2sHlfSVNik+oVsjFwiRNiguHVaQ//XHH3EUuFiapdI7cK+L0R0mT5Mh9ivo9doBjz7zo9EdJE+PIfYr6PXbAPrukiXLkPmVuyCFpGhy5S1IDGe6S1EC2ZSZs+CaqDytJmgZH7hN05+Enue3Pv3d6OV9vokqaFkfuE9IPdoDfee/POeVR0lQ5cp+QfivGYJdUBcN9gnxISVJVDHdJaiDDfQLcE1VS1Qz3kg3eSHVmjKSqGO4lcoaMpLow3EtisEuqE8O9JE59lFQnhnuJnPooqS4Md0lqIMO9BE59lFQ3hnsJ3A9VUt24cNgquR+qpDpz5L4KLuUrqe4cua+C0x4l1Z0j91WyDSOpzhy5r0C/z+52eZLqznAvaHB5gR1zF9ljl1RrhntB9tklzRJ77itgn13SrCgU7hGxKyKOR8SJiLh1xOe/FRHHIuK7EfFARFxWfqnV8QlUSbNmbLhHxBrgDuBaYDtwQ0RsHzrtO0AnM/8Z8DXgc2UXWhU335A0i4qM3K8ETmTmY5n5E+BuYPfgCZn5YGb+uHd4CNhcbpnVcI12SbOqSLhvAp4aOF7svbeUG4G/HvVBROyNiIWIWDh58mTxKitgsEuaZUXCPUa8lyNPjPgg0AF+d9TnmbkvMzuZ2dmwYUPxKivg7BhJs6zIVMhF4NKB483A08MnRcQ7gY8BP5+Z/1BOedXo30B1doykWVVk5H4E2BYRcxFxHrAHmB88ISLeAnwRuD4zny2/zOlyCV9Js25suGfmy8DNwP3Ao8C9mflIRNweEdf3Tvtd4J8AfxYRD0fE/BJfrvYctUtqgkJPqGbmAeDA0HufGHj9zpLrmrr+ujH9+eyO2iXNMpcf6OkvCNZfN8ZRu6RZZrgP2L5xHfd86Kqqy5Cks9b6cHcZX0lN1PqFwwaD3T67pKZo/cgdbMdIap5Wj9xd7VFSU7U23F3tUVKTtTbcXTtGUpO1NtzBnZUkNVerw12Smqp1s2Wc1y6pDVo3cndeu6Q2aM3IfXjE7rx2SU3WinAfnPbYXxhMkpqsFeHutEdJbdOanrvTHiW1SePD3SUGJLVR48Pd/VAltVHjwx1syUhqn8beUPVhJUlt1shwd+qjpLZrXLgPBrtTHyW1VeN67s5pl6SGhXt/2qM3UCW1XaPC3WmPktTVmHB31C5JZzQi3N0PVZJerRHh7k1USXq1RoQ7+BSqJA1qTLhLks4oFO4RsSsijkfEiYi4dcTn/ygi7ul9fjgitpZd6FJc9VGSXmvsE6oRsQa4A3gXsAgciYj5zDw2cNqNwI8y859GxB7gs8AvTaLgvv7aMf1g90aqJJ1RZPmBK4ETmfkYQETcDewGBsN9N/Cp3uuvAf8xIiIzs8RagdeGen/tGPvtknRGkXDfBDw1cLwI7FjqnMx8OSJeAC4GniujyEH9YDfUJWlpRcI9Rrw3PCIvcg4RsRfYC7Bly+pCefsb17H9jev45C/87Kp+vyS1QZFwXwQuHTjeDDy9xDmLEfE64PXAa+5yZuY+YB9Ap9NZVcvGUJek8YrMljkCbIuIuYg4D9gDzA+dMw/8Su/1+4G/nUS/XZJUzNiRe6+HfjNwP7AG+FJmPhIRtwMLmTkP/DHw1Yg4QXfEvmeSRUuSlldos47MPAAcGHrvEwOvTwH/vNzSJEmr5ROqktRAhrskNZDhLkkNZLhLUgMZ7pLUQFHVdPSIOAn8/Sp/+3omsLRBzXnN7eA1t8PZXPNlmblh3EmVhfvZiIiFzOxUXcc0ec3t4DW3wzSu2baMJDWQ4S5JDTSr4b6v6gIq4DW3g9fcDhO/5pnsuUuSljerI3dJ0jJqHe513ph7Ugpc829FxLGI+G5EPBARl1VRZ5nGXfPAee+PiIyImZ9ZUeSaI+IXez/rRyLizmnXWLYCf7a3RMSDEfGd3p/v66qosywR8aWIeDYivr/E5xERf9D77/HdiHhrqQVkZi1/0V1e+H8CPwOcB/xXYPvQOf8S+ELv9R7gnqrrnsI1XwP8497r32jDNffOuwA4CBwCOlXXPYWf8zbgO8CFveNLqq57Cte8D/iN3uvtwBNV132W1/x24K3A95f4/Drgr+nuZLcTOFzm96/zyP30xtyZ+ROgvzH3oN3An/Refw14R0SM2vJvVoy95sx8MDN/3Ds8RHdnrFlW5OcM8Gngc8CpaRY3IUWu+Sbgjsz8EUBmPjvlGstW5JoTWNd7/Xpeu+PbTMnMg4zYkW7AbuBPs+sQ8IaI2FjW969zuI/amHvTUudk5stAf2PuWVXkmgfdSPf//LNs7DVHxFuASzPzL6dZ2AQV+Tm/CXhTRHwrIg5FxK6pVTcZRa75U8AHI2KR7v4Rt0yntMqs9O/7ihTarKMipW3MPUMKX09EfBDoAD8/0Yomb9lrjohzgN8HfnVaBU1BkZ/z6+i2Zq6m+6+zv4uIKzLzf0+4tkkpcs03AF/JzN+LiKvo7u52RWa+MvnyKjHR/KrzyH0lG3Oz3MbcM6TINRMR7wQ+Blyfmf8wpdomZdw1XwBcAXwzIp6g25ucn/GbqkX/bO/PzJcy83HgON2wn1VFrvlG4F6AzHwIWEt3DZamKvT3fbXqHO5t3Jh77DX3WhRfpBvss96HhTHXnJkvZOb6zNyamVvp3me4PjMXqim3FEX+bN9H9+Y5EbGebpvmsalWWa4i1/wk8A6AiLicbrifnGqV0zUP/IverJmdwAuZ+UxpX73qO8pj7jZfB/x3unfZP9Z773a6f7mh+8P/M+AE8G3gZ6queQrX/DfA/wIe7v2ar7rmSV/z0LnfZMZnyxT8OQfw74FjwPeAPVXXPIVr3g58i+5MmoeBd1dd81le713AM8BLdEfpNwK/Dvz6wM/4jt5/j++V/efaJ1QlqYHq3JaRJK2S4S5JDWS4S1IDGe6S1ECGuyQ1kOGuVouI34yIRyPiP1ddi1Qmp0Kq1SLivwHXZvcp0HHnrsnMn06hLOmsOXJXa0XEF+guQTsfES9ExFcj4m8j4n9ExE29c67urTF+J90HTaSZ4MhdrdZbr6YD3Ay8l+7aNefTXUt9B93H/v8KuKLI6F6qC0fu0hn7M/P/ZeZzwIN01yAH+LbBrlljuEtnDP8ztn/8f6ddiHS2DHfpjN0RsTYiLqa7jvqRiuuRVs1wl874Nt3++iHg05k509u8qd28oSoBEfEp4P9k5uerrkUqgyN3SWogR+6S1ECO3CWpgQx3SWogw12SGshwl6QGMtwlqYEMd0lqoP8PqdskWuv9em8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.4 对于正例有着较强预测能力的ROC曲线"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "阈值为0.5时，我们让正例的预测值有95%的可能性是正确预测，负例的预测值有70%的可能性是正确预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(200)\n",
    "\n",
    "def f(x):\n",
    "    if x == 1:\n",
    "        if rand() > 0.05:\n",
    "            return rand() / 2 + 0.5\n",
    "        else:\n",
    "            return rand() / 2\n",
    "    else:\n",
    "        if rand() > 0.3:\n",
    "            return rand() / 2\n",
    "        else:\n",
    "            return rand() / 2 + 0.5\n",
    "        \n",
    "y_pred = np.array([f(yi) for yi in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。可以看出ROC曲线的形状是偏上的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.826.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9f34d30>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFGVJREFUeJzt3X+MZWV9x/H3lwW6KQVlf5Bsd4Ed0yVhxAbMhF1CgmykhF0jE4xVqKS1IYAWUKJtQpGgYqJBaU1MaHXTqtV0+aGJ7GpXqUXMNoZdGQKCu3TbLeAyQMoAFtJalB/f/nHvwOXu/Dg7c+4995z7fiWTzL332ZnvYWY/PPu9z3meyEwkSc1yWNUFSJLKZ7hLUgMZ7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ10eFXfeMWKFbl27dqqvr0k1dJ99933TGaunG9cZeG+du1aJiYmqvr2klRLEfGLIuNsy0hSAxnuktRAhrskNVBlPfeZvPTSS0xOTvLiiy9WXcqiLV26lDVr1nDEEUdUXYqkITRQ4T45OcnRRx/N2rVriYiqy1mwzOTZZ59lcnKSkZGRqsuRNITmbctExFcj4umI+Pksr0dEfCki9kfEgxHx9oUW8+KLL7J8+fJaBztARLB8+fJG/AtEUj0V6bl/HThvjtc3AevaH5cBf7uYguoe7NOach2S6mnetkxm7oyItXMMGQe+ka3z+nZFxJsjYlVmPlVSjZJUO1t3H2DbA0/M+Nro7x7DJ9/91p5+/zJ67quBxzseT7afOyjcI+IyWrN7TjjhhBK+tST13lxBPZvdjz4HwPqRZb0oaV5lhPtM/YcZT93OzC3AFoCxsbGBP5k7M8lMDjvMFaNS3S0koKctJKjXjyxj/NTV/NH6aiayZYT7JHB8x+M1wJMlfN1KPPbYY2zatImNGzdyzz33cPXVV3PTTTeRmbzrXe/ixhtvBOAHP/gB1157La+88gorVqzgrrvuqrhyqbcWE46DYDEz6aqDeiHKCPftwJURcSuwHni+jH77p7+7h71PvrDo4joV7XPt27ePr33ta1x33XVs2LCB++67j2OPPZZzzz2XO+64gzPPPJNLL72UnTt3MjIywnPPPVdqnVIvLDacq24zLFYdA3ox5g33iLgFOBtYERGTwCeBIwAy88vADmAzsB/4FfCnvSq2X0488UQ2bNjAtm3bOPvss1m5srUB2wc+8AF27tzJkiVLOOuss15bw75sWT1/2bUwdZ3BLjachy0c667IapmL5nk9gStKq6it1+8kz+Woo44CWj33mWSmSx2HwGwhXtcZrOE8XAbqDtVBs379ej760Y/yzDPPcOyxx3LLLbdw1VVXccYZZ3DFFVfw6KOPvtaWcfZerV7MpmcLcUNSdWC4z2HVqlV87nOfY+PGjWQmmzdvZnx8HIAtW7bwnve8h1dffZXjjjuOH/7whxVXOxz6OZs2xFVnMVvrodfGxsay+7COhx9+mJNPPrmSenqhaddzqPo5mwYMYg2FiLgvM8fmG+fMXTMqI5idTUvVMdw1o20PPMHep15gdNUxC/4aBrFUnYEL96asRKmq3VWm0VXHcNvlZ1RdhqQFGKj76pcuXcqzzz5b+2Cc3s996dKlVZciaUgN1Mx9zZo1TE5OMjU1VXUpizZ9EpMkVWGgwv2II47w5CJJKsFAtWUkSeUw3HWQrbsPvLaMUVI9Ge46yPT69vFTV1dciaSFMtw1o/Ujy1yfLtWY4a43sCUjNYPhrjewJSM1g+Gu10zP2m3JSPVnuAtoBfu133kIcNYuNYHhLuD1dsxnL3ibs3apAQx3vcZ2jNQchrskNZDhLkkNNFAbh6n/pk9cWuzBHJIGi+E+xDpXyEyfmiSpGQz3IdUZ7K6QkZrHnvuQcumj1GyG+xBz6aPUXIa7JDWQ4S5JDWS4DyG39ZWaz3AfQm7rKzVfoXCPiPMiYl9E7I+Ia2Z4/YSIuDsi7o+IByNic/mlqky+mSo127zhHhFLgJuBTcAocFFEjHYNuw64PTNPAy4E/qbsQlUOWzLScCgycz8d2J+Zj2Tmb4BbgfGuMQlM37v+JuDJ8kpUmWzJSMOhyB2qq4HHOx5PAuu7xnwK+OeIuAo4CjinlOpUKk9akoZHkZl7zPBcdj2+CPh6Zq4BNgPfjIiDvnZEXBYRExExMTU1dejValGctUvDo0i4TwLHdzxew8Ftl0uA2wEy8x5gKbCi+wtl5pbMHMvMsZUrVy6sYi2Ks3ZpOBQJ93uBdRExEhFH0nrDdHvXmAPAOwEi4mRa4e7UXJIqMm+4Z+bLwJXAncDDtFbF7ImIGyLi/PawjwOXRsTPgFuAD2Zmd+tGFXKVjDRcCm35m5k7gB1dz13f8fle4MxyS1OZ7LdLw8U7VIeAq2Sk4WO4DwFn7dLw8SSmBus8H9VZuzRcnLk3WOfB187apeHizL2hOvvst11+RtXlSOozZ+4NZZ9dGm6GewO5OkaS4d4wW3cf4NrvPAQ4a5eGmeHeIJ3B/tkL3uasXRpihnuDTPfZDXZJhntD2GeX1MlwbwD77JK6Ge4NYDtGUjfDvSFsx0jqZLjXnPu0S5qJ4V5z3okqaSaGe425QkbSbNw4rIamt/Kdbsc4a5fUzXCvoc492sdPXe2sXdJBDPeacStfSUXYc68Z30CVVIQz95rwyDxJh8KZe014ZJ6kQ+HMvUZGVx1jn11SIc7cJamBDHdJaiDDvQbcP0bSobLnPsC8E1XSQhnuA8w7USUtlOE+4FwhI2kh7LlLUgMVCveIOC8i9kXE/oi4ZpYx74uIvRGxJyK2llumJOlQzBvuEbEEuBnYBIwCF0XEaNeYdcBfAmdm5luBq3tQ61BxhYykxSgycz8d2J+Zj2Tmb4BbgfGuMZcCN2fmLwEy8+lyyxwuW3cf4NrvPAS4QkbSwhQJ99XA4x2PJ9vPdToJOCkifhIRuyLivJm+UERcFhETETExNTW1sIqHwPTOj5+94G2ukJG0IEXCPWZ4LrseHw6sA84GLgL+LiLefNAfytySmWOZObZy5cpDrXUoeHSepDIUCfdJ4PiOx2uAJ2cYsy0zX8rMR4F9tMJeh8j92iWVoUi43wusi4iRiDgSuBDY3jXmDmAjQESsoNWmeaTMQoeJs3ZJizVvuGfmy8CVwJ3Aw8DtmbknIm6IiPPbw+4Eno2IvcDdwF9k5rO9KlqSNLdCd6hm5g5gR9dz13d8nsDH2h+SpIp5h6okNZDhLkkN5MZhA6LzAOzRVcdUXY6kmnPmPiA8AFtSmZy5D4DOG5fc3ldSGZy5DwBvXJJUNsO9Ym43IKkXDPcKufujpF4x3CvSGezu/iipbIZ7RdzWV1IvGe4Vss8uqVcMd0lqIMO9Ap6PKqnXDPc+c4WMpH4w3PvMN1Il9YPhXgHfSJXUa4a7JDWQ4d5HvpEqqV8M9z5ygzBJ/WK494kbhEnqJ8O9T5y1S+onw70PnLVL6jfDvQ+ctUvqN8O9x5y1S6qC4d5DbjUgqSqGe494GIekKhnuPWCwS6qa4V4yg13SIDDcS+auj5IGgeFeIlfGSBoUhcI9Is6LiH0RsT8irplj3HsjIiNirLwSB9/W3Qd4/1fucWWMpIFx+HwDImIJcDPwB8AkcG9EbM/MvV3jjgY+AuzuRaGDbNsDT7D3qRdYP7KM8VNXO2uXVLl5wx04HdifmY8ARMStwDiwt2vcZ4DPA39eaoUDrrMVc9vlZ1RdjiQBxdoyq4HHOx5Ptp97TUScBhyfmd8rsbZacGsBSYOoSLjHDM/lay9GHAZ8Efj4vF8o4rKImIiIiampqeJVDjjfQJU0aIqE+yRwfMfjNcCTHY+PBk4BfhwRjwEbgO0zvamamVsycywzx1auXLnwqgeEJytJGlRFwv1eYF1EjETEkcCFwPbpFzPz+cxckZlrM3MtsAs4PzMnelLxALElI2lQzRvumfkycCVwJ/AwcHtm7omIGyLi/F4XOOhsyUgaREVWy5CZO4AdXc9dP8vYsxdf1uDrXCUjSYPGO1QXyJaMpEFmuC+CLRlJg8pwl6QGMtwXwCWQkgad4b4A9tslDTrDfYHst0saZIb7IbIlI6kODPdD0HmEni0ZSYPMcD8EHqEnqS4M90Nkr11SHRjuktRAhrskNZDhLkkNZLgX5BJISXViuBfkXamS6qTQfu7DbOvuA2x74An2PvWCK2Uk1YbhPofOm5bWjyxz1i6pNgz3WXQGuzctSaobe+6z8G5USXVmuM/BHrukujLcZ+CyR0l1Z7jPwGWPkurOcJ+FLRlJdWa4S1IDGe5d7LdLagLXubdN34k6Hez22yXVmeHe1rnFwPipq+23S6o1w73D6KpjuO3yM6ouQ5IWzZ67JDWQ4Y5vokpqHsMdb1qS1DyFwj0izouIfRGxPyKumeH1j0XE3oh4MCLuiogTyy+1fFt3H+D9X7nHvdolNc684R4RS4CbgU3AKHBRRIx2DbsfGMvM3we+DXy+7EJ7YXqFzOiqY5y1S2qUIqtlTgf2Z+YjABFxKzAO7J0ekJl3d4zfBVxcZpG9MN1nXz+yzBUykhqnSFtmNfB4x+PJ9nOzuQT4/kwvRMRlETERERNTU1PFqyxZ50EcztglNVGRcI8ZnssZB0ZcDIwBX5jp9czckpljmTm2cuXK4lWWzIM4JDVdkbbMJHB8x+M1wJPdgyLiHOATwDsy89fllFe+znaMwS6pqYrM3O8F1kXESEQcCVwIbO8cEBGnAV8Bzs/Mp8svszwue5Q0DOYN98x8GbgSuBN4GLg9M/dExA0RcX572BeA3wG+FREPRMT2Wb7cQHDWLqnpCu0tk5k7gB1dz13f8fk5JdclSVqEodk4bHpL3+l17ZLUZEOz/YA3LEkaJo2fuXfP2L1hSdIwaPzM3Rm7pGHU+Jk7eAiHpOHT+Jm7JA2jxs7cXR0jaZg1duZur13SMGvkzN3tfCUNu8bN3N3OV5IaGO5u5ytJDQx3cGMwSWpUuE/32iVp2DUq3N2rXZJaGhXuYEtGkqBB4W5LRpJe15hwtyUjSa9rTLiDLRlJmtaocJcktTQi3O23S9IbNSLc7bdL0hs1ItzBfrskdWpMuEuSXlf7cLffLkkHq3W4u72vJM2stuHeGexu7ytJb1TbcHffdkmaXS3DvfMYPYNdkg5Wy3B3Xbskza2W4Q6ua5ekuRQK94g4LyL2RcT+iLhmhtd/KyJua7++OyLWll2oJKm4ecM9IpYANwObgFHgoogY7Rp2CfDLzPw94IvAjWUXKkkqrsjM/XRgf2Y+kpm/AW4FxrvGjAP/0P7828A7IyLKK1OSdCiKhPtq4PGOx5Pt52Yck5kvA88Dy8sosNunv7vHO1IlaR5Fwn2mGXguYAwRcVlETETExNTUVJH6ZrR+ZJkrZSRpDocXGDMJHN/xeA3w5CxjJiPicOBNwEHT68zcAmwBGBsbOyj8i/jku9+6kD8mSUOlyMz9XmBdRIxExJHAhcD2rjHbgT9pf/5e4EeZuaDwliQt3rwz98x8OSKuBO4ElgBfzcw9EXEDMJGZ24G/B74ZEftpzdgv7GXRkqS5FWnLkJk7gB1dz13f8fmLwB+WW5okaaFqe4eqJGl2hrskNZDhLkkNZLhLUgMZ7pLUQFHVcvSImAJ+scA/vgJ4psRy6sBrHg5e83BYzDWfmJkr5xtUWbgvRkRMZOZY1XX0k9c8HLzm4dCPa7YtI0kNZLhLUgPVNdy3VF1ABbzm4eA1D4eeX3Mte+6SpLnVdeYuSZrDQIf7MB7MXeCaPxYReyPiwYi4KyJOrKLOMs13zR3j3hsRGRG1X1lR5Joj4n3tn/WeiNja7xrLVuB3+4SIuDsi7m//fm+uos6yRMRXI+LpiPj5LK9HRHyp/d/jwYh4e6kFZOZAftDaXvg/gbcARwI/A0a7xvwZ8OX25xcCt1Vddx+ueSPw2+3PPzwM19wedzSwE9gFjFVddx9+zuuA+4Fj24+Pq7ruPlzzFuDD7c9HgceqrnuR13wW8Hbg57O8vhn4Pq2T7DYAu8v8/oM8cx/Gg7nnvebMvDszf9V+uIvWyVh1VuTnDPAZ4PPAi/0srkeKXPOlwM2Z+UuAzHy6zzWWrcg1J3BM+/M3cfCJb7WSmTuZ4US6DuPAN7JlF/DmiFhV1vcf5HAfqIO5+6TINXe6hNb/+ets3muOiNOA4zPze/0srIeK/JxPAk6KiJ9ExK6IOK9v1fVGkWv+FHBxREzSOj/iqv6UVplD/ft+SAod1lGR0g7mrpHC1xMRFwNjwDt6WlHvzXnNEXEY8EXgg/0qqA+K/JwPp9WaOZvWv87+NSJOycz/7nFtvVLkmi8Cvp6ZfxURZ9A63e2UzHy19+VVoqf5Ncgz90M5mJu5DuaukSLXTEScA3wCOD8zf92n2nplvms+GjgF+HFEPEarN7m95m+qFv3d3paZL2Xmo8A+WmFfV0Wu+RLgdoDMvAdYSmsPlqYq9Pd9oQY53IfxYO55r7ndovgKrWCvex8W5rnmzHw+M1dk5trMXEvrfYbzM3OimnJLUeR3+w5ab54TEStotWke6WuV5SpyzQeAdwJExMm0wn2qr1X213bgj9urZjYAz2fmU6V99arfUZ7n3ebNwL/Tepf9E+3nbqD1lxtaP/xvAfuBnwJvqbrmPlzzvwD/BTzQ/thedc29vuausT+m5qtlCv6cA/hrYC/wEHBh1TX34ZpHgZ/QWknzAHBu1TUv8npvAZ4CXqI1S78E+BDwoY6f8c3t/x4Plf177R2qktRAg9yWkSQtkOEuSQ1kuEtSAxnuktRAhrskNZDhrqEWER+JiIcj4h+rrkUqk0shNdQi4t+ATdm6C3S+sUsy85U+lCUtmjN3Da2I+DKtLWi3R8TzEfHNiPhRRPxHRFzaHnN2e4/xrbRuNJFqwZm7hlp7v5ox4ErgAlp71xxFay/19bRu+/8n4JQis3tpUDhzl163LTP/LzOfAe6mtQc5wE8NdtWN4S69rvufsdOP/7ffhUiLZbhLrxuPiKURsZzWPur3VlyPtGCGu/S6n9Lqr+8CPpOZtT7mTcPNN1QlICI+BfxPZt5UdS1SGZy5S1IDOXOXpAZy5i5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSA/0/jovdD/w+aPIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.5 对于负例有着较强预测能力的ROC曲线"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "阈值为0.5时，我们让正例的预测值有70%的可能性是正确预测，负例的预测值有95%的可能性是正确预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(120)\n",
    "\n",
    "def f(x):\n",
    "    if x == 1:\n",
    "        if rand() > 0.3:\n",
    "            return rand() / 2 + 0.5\n",
    "        else:\n",
    "            return rand() / 2\n",
    "    else:\n",
    "        if rand() > 0.05:\n",
    "            return rand() / 2\n",
    "        else:\n",
    "            return rand() / 2 + 0.5\n",
    "        \n",
    "y_pred = np.array([f(yi) for yi in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。可以看出ROC曲线的形状是偏左的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.830.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9f04198>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAE6dJREFUeJzt3X+s3XV9x/HnmwJrxkB+tCRdf9CalYQrLmhuaBsSaCOSghk3GmcKms2FALqhcZolDAkqJjoUZ2ZC1M4halZ+aDJaXZU5RLoYWltih7asW0exXCDj8mOQzVV+vffHOReOt+fe8733np+f83wkN5zvOZ/e8/5w7n3108/38/1+IjORJJXlmF4XIElqP8NdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVKBje/XGixYtypUrV/bq7SVpID344INPZ+biVu16Fu4rV65kz549vXp7SRpIEfHLKu2clpGkAhnuklQgw12SCtSzOfdmXnrpJcbHxzly5EivS5m3hQsXsmzZMo477rhelyJpCPVVuI+Pj3PiiSeycuVKIqLX5cxZZvLMM88wPj7OqlWrel2OpCHUclomIm6NiKci4hfTvB4R8aWIOBgRD0XEW+dazJEjRzjttNMGOtgBIoLTTjutiH+BSBpMVebcbwM2zvD6xcDq+tdVwJfnU9CgB/ukUvohaTC1nJbJzB0RsXKGJmPAN7O2X9/OiDg5IpZk5pNtqlGSBt6WXYfZuvdxAEZ+9yQ+8Qdv6uj7tWPOfSnwWMPxeP25o8I9Iq6iNrpnxYoVbXhrSeoPjeHdzK5DzwKwZtWpXamnHeHebP6h6a7bmbkZ2AwwOjra9ztzZyaZyTHHuGJU0swB3iq816w6lbFzlnL5mu4MbNsR7uPA8objZcATbfi+PfHoo49y8cUXs2HDBh544AE+8pGPcPPNN5OZvOMd7+Cmm24C4Ac/+AHXXXcdr7zyCosWLeLee+/tceWS2qlZkM8U4N0O71baEe7bgGsi4g5gDfB8O+bbP/Xdfex/4oV5F9eo6jzXgQMH+PrXv87111/P2rVrefDBBznllFO46KKLuPvuuznvvPO48sor2bFjB6tWreLZZ59ta52Sume60XizIO+3AJ9Jy3CPiNuB9cCiiBgHPgEcB5CZXwG2A5cAB4FfAX/SqWK75YwzzmDt2rVs3bqV9evXs3hx7QZs733ve9mxYwcLFizg/PPPf20N+6mndmcOTVL7TIb6dKPxQQryZqqslrmsxesJ/FnbKqrr9JnkmZxwwglAbc69mcx0qaM0YKaO0BtDfZBDfDqeKZzBmjVruP/++3n66ad55ZVXuP3227ngggtYt24d999/P4cOHQJwWkbqc1t2Hea6f/j5a4EOtVD/zDvfzJ1Xrysu2KHPbj/Qb5YsWcJnP/tZNmzYQGZyySWXMDY2BsDmzZt517vexauvvsrpp5/OD3/4wx5XK2k6kyP2z7zzzUUGeTMx3dRDp42OjubUzToefvhhzjrrrJ7U0wml9UfqN63Wlk/a/+QLjCw5iTuvXteFqjorIh7MzNFW7Ry5S+pLVYK76oVBI0tOYuycpW2rbRAY7pL6wkwnPKdT6snQdui7cC9lJUqvprukQdIY6FPD3OCen74K94ULF/LMM88M/G1/J+/nvnDhwl6XIvWFKhcKGebt1VfhvmzZMsbHx5mYmOh1KfM2uROTNMxKv1Con/VVuB933HHuXCQVoFmoG+Ld1VfhLmlwVL1DoqHeG4a7JKD6mvFJg3SHxGFkuEtDrtW8+HQM8P5muEtDbPKeK2BYl8Zwlwozm+mVydH6MN1zZVgY7lJhtu59/LV7qbTiaL1chrvUR2Z7UrOZkm6Spbnzfu5SH5kcdc/HMN4kS0dz5C71gckRu6NutYvhLvXIdDfNctStdjDcpS5oNpfuTbPUSYa71EEzXSBkoKuTDHepormsZPEeK+oVw12awUybSVRhqKtXDHdpGlMvzTeoNUgMdw2F+UypeGm+BpHhruJNHYFX5Uhdg8xwV9Eag90RuIaJ4a6BV2VHIINdw8Zw18CZGubuCCQdzXBXX2t1Zefkfw1w6TdVCveI2Aj8DbAA+Fpm/tWU11cA3wBOrre5NjO3t7lWFa5KkE8+NsylmbUM94hYANwCvB0YB3ZHxLbM3N/Q7Hrgrsz8ckSMANuBlR2oV4VpdZGQQS7NTZWR+7nAwcx8BCAi7gDGgMZwT2By25c3AE+0s0gNrlbry715ltQZVcJ9KfBYw/E4sGZKm08C/xQRHwJOAC5sS3UaaFXWlxvoUmdUCfdo8lxOOb4MuC0zvxAR64BvRcTZmfnqb3yjiKuAqwBWrPCXeVDMdes3lyFKvVMl3MeB5Q3Hyzh62uUKYCNAZj4QEQuBRcBTjY0yczOwGWB0dHTqXxDqE7NZajgTR+VS71QJ993A6ohYBTwObAIun9LmMPA24LaIOAtYCEy0s1B11kwnNg1pafC0DPfMfDkirgHuobbM8dbM3BcRNwJ7MnMb8DHgbyPiz6lN2bw/Mx2ZDwjvfiiVp9I69/qa9e1Tnruh4fF+4Lz2lqZOajZSd25cKodXqA6R6aZeHKlL5THcC+WGzNJwM9wL4tWekiYZ7gPKkbmkmRjuA6LK2nMDXdIkw71PtQpzg1zSTAz3BnO9zL4TDHNJ8zH04d7qJGSvGOaS5mMow9313pJKNzThbqBLGiZDE+5b9z7O/idfYGTJSQa6pOINTbgDjCw5iTuvXtfrMiSp447pdQHdsGXX4demYiRpGAxFuE/OtY+ds7THlUhSdwxFuEPt5Klz7JKGxdCEuyQNk+LD3fl2ScOo+HB3vl3SMCo+3MH5dknDZyjCXZKGjeEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVqFK4R8TGiDgQEQcj4tpp2rwnIvZHxL6I2NLeMiVJs9Fyg+yIWADcArwdGAd2R8S2zNzf0GY18JfAeZn5XESc3qmCJUmtVRm5nwsczMxHMvNF4A5gbEqbK4FbMvM5gMx8qr1lSpJmo0q4LwUeazgerz/X6EzgzIj4SUTsjIiNzb5RRFwVEXsiYs/ExMTcKpYktVQl3KPJcznl+FhgNbAeuAz4WkScfNQfytycmaOZObp48eLZ1ipJqqhKuI8DyxuOlwFPNGmzNTNfysxDwAFqYS9J6oEq4b4bWB0RqyLieGATsG1Km7uBDQARsYjaNM0j7SxUklRdy3DPzJeBa4B7gIeBuzJzX0TcGBGX1pvdAzwTEfuB+4C/yMxnOlW0JGlmLZdCAmTmdmD7lOduaHicwEfrX5KkHvMKVUkqUNHhvmXXYXYderbXZUhS1xUd7lv3Pg7A2DlTl+VLUtmKDneANatO5fI1K3pdhiR1VfHhLknDyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFajYcPe+MpKGWbHh7n1lJA2zYsMdvK+MpOFVdLhL0rAy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBigx3r06VNOyKDHevTpU07IoMd/DqVEnDrdhwl6RhZrhLUoEMd0kqkOEuSQWqFO4RsTEiDkTEwYi4doZ2746IjIjR9pU4Oy6DlKQK4R4RC4BbgIuBEeCyiBhp0u5E4MPArnYXORsug5SkaiP3c4GDmflIZr4I3AGMNWn3aeBzwJE21jcnLoOUNOyqhPtS4LGG4/H6c6+JiLcAyzPze22sTZI0R1XCPZo8l6+9GHEM8EXgYy2/UcRVEbEnIvZMTExUr1KSNCtVwn0cWN5wvAx4ouH4ROBs4McR8SiwFtjW7KRqZm7OzNHMHF28ePHcq5YkzahKuO8GVkfEqog4HtgEbJt8MTOfz8xFmbkyM1cCO4FLM3NPRyqWJLXUMtwz82XgGuAe4GHgrszcFxE3RsSlnS5QkjR7x1ZplJnbge1Tnrthmrbr51+WJGk+vEJVkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklSgSpt1DIItuw6zde/j7H/yBUaWnNTrciSpp4oZuTcG+9g5S3tdjiT1VDEjd4CRJSdx59Xrel2GJPVcMSN3SdLrDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAlcI9IjZGxIGIOBgR1zZ5/aMRsT8iHoqIeyPijPaXKkmqqmW4R8QC4BbgYmAEuCwiRqY0+xkwmpm/D3wH+Fy7C5UkVVdl5H4ucDAzH8nMF4E7gLHGBpl5X2b+qn64E1jW3jIlSbNRJdyXAo81HI/Xn5vOFcD3m70QEVdFxJ6I2DMxMVG9SknSrFQJ92jyXDZtGPE+YBT4fLPXM3NzZo5m5ujixYurVylJmpUqOzGNA8sbjpcBT0xtFBEXAh8HLsjMX7enPEnSXFQZue8GVkfEqog4HtgEbGtsEBFvAb4KXJqZT7W/TEnSbLQM98x8GbgGuAd4GLgrM/dFxI0RcWm92eeB3wG+HRF7I2LbNN+uI7bsOsyuQ8928y0lqa9V2iA7M7cD26c8d0PD4wvbXNesbN37OABj58x0nleShkcxV6iuWXUql69Z0esyJKkvFBPukqTXGe6SVCDDXZIKNPDh7koZSTrawIe7K2Uk6WgDHe6To3ZXykjSbxrocHfULknNDXS4g+vbJamZgQ93SdLRDHdJKpDhLkkFMtwlqUCGuyQVaGDD3StTJWl6AxvurnGXpOkNbLiDa9wlaToDHe6SpOYMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklSggQx3r06VpJkNZLh7daokzWwgwx28OlWSZjKw4S5Jmp7hLkkFMtwlqUCVwj0iNkbEgYg4GBHXNnn9tyLizvrruyJiZbsLlSRV1zLcI2IBcAtwMTACXBYRI1OaXQE8l5m/B3wRuKndhUqSqqsycj8XOJiZj2Tmi8AdwNiUNmPAN+qPvwO8LSKifWVKkmajSrgvBR5rOB6vP9e0TWa+DDwPnNaOAqf61Hf3eQGTJLVQJdybjcBzDm2IiKsiYk9E7JmYmKhSX1NrVp3qBUySNINjK7QZB5Y3HC8DnpimzXhEHAu8AThqeJ2Zm4HNAKOjo0eFfxWf+IM3zeWPSdJQqTJy3w2sjohVEXE8sAnYNqXNNuCP64/fDfwoM+cU3pKk+Ws5cs/MlyPiGuAeYAFwa2bui4gbgT2ZuQ34O+BbEXGQ2oh9UyeLliTNrMq0DJm5Hdg+5bkbGh4fAf6wvaVJkubKK1QlqUCGuyQVyHCXpAIZ7pJUIMNdkgoUvVqOHhETwC/n+McXAU+3sZxBYJ+Hg30eDvPp8xmZubhVo56F+3xExJ7MHO11Hd1kn4eDfR4O3eiz0zKSVCDDXZIKNKjhvrnXBfSAfR4O9nk4dLzPAznnLkma2aCO3CVJM+jrcB/Gjbkr9PmjEbE/Ih6KiHsj4oxe1NlOrfrc0O7dEZERMfArK6r0OSLeU/+s90XElm7X2G4VfrZXRMR9EfGz+s/3Jb2os10i4taIeCoifjHN6xERX6r//3goIt7a1gIysy+/qN1e+D+BNwLHA/8KjExp86fAV+qPNwF39rruLvR5A/Db9ccfHIY+19udCOwAdgKjva67C5/zauBnwCn149N7XXcX+rwZ+GD98QjwaK/rnmefzwfeCvximtcvAb5PbSe7tcCudr5/P4/ch3Fj7pZ9zsz7MvNX9cOd1HbGGmRVPmeATwOfA450s7gOqdLnK4FbMvM5gMx8qss1tluVPidwUv3xGzh6x7eBkpk7aLIjXYMx4JtZsxM4OSKWtOv9+znc+2pj7i6p0udGV1D7m3+QtexzRLwFWJ6Z3+tmYR1U5XM+EzgzIn4SETsjYmPXquuMKn3+JPC+iBintn/Eh7pTWs/M9vd9Vipt1tEjbduYe4BU7k9EvA8YBS7oaEWdN2OfI+IY4IvA+7tVUBdU+ZyPpTY1s57av87+JSLOzsz/7nBtnVKlz5cBt2XmFyJiHbXd3c7OzFc7X15PdDS/+nnkPpuNuZlpY+4BUqXPRMSFwMeBSzPz112qrVNa9flE4GzgxxHxKLW5yW0DflK16s/21sx8KTMPAQeohf2gqtLnK4C7ADLzAWAhtXuwlKrS7/tc9XO4D+PG3C37XJ+i+Cq1YB/0eVho0efMfD4zF2XmysxcSe08w6WZuac35bZFlZ/tu6mdPCciFlGbpnmkq1W2V5U+HwbeBhARZ1EL94muVtld24A/qq+aWQs8n5lPtu279/qMcouzzZcA/07tLPvH68/dSO2XG2of/reBg8BPgTf2uuYu9Pmfgf8C9ta/tvW65k73eUrbHzPgq2Uqfs4B/DWwH/g5sKnXNXehzyPAT6itpNkLXNTrmufZ39uBJ4GXqI3SrwA+AHyg4TO+pf7/4+ft/rn2ClVJKlA/T8tIkubIcJekAhnuklQgw12SCmS4S1KBDHcNtYj4cEQ8HBF/3+tapHZyKaSGWkT8G3Bx1q4CbdV2QWa+0oWypHlz5K6hFRFfoXYL2m0R8XxEfCsifhQR/xERV9bbrK/fY3wLtQtNpIHgyF1DrX6/mlHgGuCd1O5dcwK1e6mvoXbZ/z8CZ1cZ3Uv9wpG79Lqtmfl/mfk0cB+1e5AD/NRg16Ax3KXXTf1n7OTx/3a7EGm+DHfpdWMRsTAiTqN2H/XdPa5HmjPDXXrdT6nNr+8EPp2ZA73Nm4abJ1QlICI+CfxPZt7c61qkdnDkLkkFcuQuSQVy5C5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIK9P/B/T61QhaoPgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.6 预测值集中在0.5左右的ROC曲线\n",
    "阈值为0.5时，我们让预测值有80%的可能性是正确预测，但值集中在0.5左右"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(220)\n",
    "\n",
    "def f(x):\n",
    "    if x == 1:\n",
    "        if rand() > 0.2:\n",
    "            return rand() / 10 + 0.5\n",
    "        else:\n",
    "            return 0.5 - rand() / 10\n",
    "    else:\n",
    "        if rand() > 0.2:\n",
    "            return 0.5 - rand() / 10\n",
    "        else:\n",
    "            return rand() / 10 + 0.5\n",
    "        \n",
    "y_pred = np.array([f(yi) for yi in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.804.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x8d264a8>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFKdJREFUeJzt3X+s3XV9x/Hnm4JrxkB+tCRdC/SalYQrLGhuaAkJ0ogEMNJgnIKazYUAOkCNbglDgooJDmUzMyFK4wA1Kz80gVZXZQ4hXQhUSkSwZd06wHKhGZcfg2yu8uu9P865cDi9t+fbe7/nfM/5fp+PpMn5nvPpve8Pt33x6ef7+X4+kZlIkuplv6oLkCSVz3CXpBoy3CWphgx3Saohw12Sashwl6QaMtwlqYYMd0mqIcNdkmpo/6q+8aJFi3L58uVVfXtJGkkPPvjgs5m5uFe7ysJ9+fLlbNmypapvL0kjKSJ+U6Sd0zKSVEOGuyTVkOEuSTVU2Zz7TF555RUmJyfZvXt31aXM28KFC1m2bBkHHHBA1aVIaqChCvfJyUkOOuggli9fTkRUXc6cZSbPPfcck5OTjI2NVV2OpAbqOS0TETdExDMR8etZPo+I+GZE7IiIhyPi3XMtZvfu3Rx++OEjHewAEcHhhx9ei3+BSBpNRebcbwLO2MvnZwIr2r8uBL41n4JGPdin1aUfkkZTz2mZzNwUEcv30mQN8L1sndd3f0QcEhFLMnNXSTVK0khat3kn6x96ao/3x//wYL74gXf29XuXMee+FHiy43qy/d4e4R4RF9Ia3XPUUUeV8K0lqXqzhfjmx58HYOXYYYMuqZRwn2n+YcZTtzNzLbAWYGJiYuhP5s5MMpP99nPFqNRUswV3p9lCfOXYYaw5YSkfXTn4wWwZ4T4JHNlxvQx4uoSvW4knnniCM888k9WrV3Pffffx2c9+lmuvvZbM5P3vfz/XXHMNAD/96U+5/PLLee2111i0aBF33XVXxZVL6qVIUHcrMvquMsRnU0a4bwAuiYhbgJXAi2XMt3/5R1vZ9vRL8y6uU9F5ru3bt3PjjTdyxRVXsGrVKh588EEOPfRQTj/9dO644w5OPvlkLrjgAjZt2sTY2BjPP/98qXVKmp8yp0mGMbiL6BnuEXEzcCqwKCImgS8CBwBk5reBjcBZwA7gt8Cf96vYQTn66KNZtWoV69ev59RTT2Xx4tYGbB/72MfYtGkTCxYs4JRTTnljDfthhw1+Pk1qijJH26Ma1HNRZLXMeT0+T+Di0ipq6/ed5L058MADgdac+0wy06WO0gCs27yTy29/BGjGaLtMQ/WE6rBZuXIln/nMZ3j22Wc59NBDufnmm7n00ks56aSTuPjii3n88cffmJZx9C6Vo3OkPj0Cv/qc4xsd1HNhuO/FkiVL+OpXv8rq1avJTM466yzWrFkDwNq1a/ngBz/I66+/zhFHHMHPfvaziquVRs9MUy6dUyqOwOcuZpt66LeJiYnsPqzj0Ucf5dhjj62knn6oW3+k+eoO89nmxg302UXEg5k50audI3dJA7P+oafYtuslxpccDDg33k+Gu6S+6R6pTwf7rRedVGFVzTB0j15WNU1Utrr0Q5qr6ZUu01MvAONLDmbNCUsrrKo5hmrkvnDhQp577rmR3/Z3ej/3hQsXVl2KNFCudBkeQxXuy5YtY3JykqmpqapLmbfpk5ikOij6IJErXYbHUIX7AQcc4MlF0oDNZ2Osbgb68BiqcJfUP/PZb8XQHj2Gu1Rz06HufivNYrhLQ24uG2d16gx1Q7w5DHdpSPUacRdlqDeT4S5VaG+jckfcmg/DXRqwmdaCzzQqN9Q1H4a7NE/7OifuWnANguEu9dArvPd1TtxA1yAY7tIMik6dTL9vWGvYGO5qrKI3Mw1vjSLDXbVQ5iHK0+8Z6BplhrtqofsQiCIMcNWZ4a6htS+jcQ+BkN7KcNdQ6HVQci8eAiG9leGugZjLckKnTaS5M9xVmqKrT2ZikEvlMtw1b0U2uDK8pcEy3NXTvkypGODScDDcG2pfVqI4pSKNHsO9hso8E3O6jeEtjRbDfYTM5QT62RjYUr0VCveIOAP4e2AB8J3M/Juuz48Cvgsc0m5zWWZuLLnWRlu3eSeX3/4I4An0knrrGe4RsQC4DngfMAk8EBEbMnNbR7MrgNsy81sRMQ5sBJb3od5G6gz2q8853tCW1NN+BdqcCOzIzMcy82XgFmBNV5sEpjf1eDvwdHklanoqxmCXVFSRaZmlwJMd15PAyq42XwL+OSIuBQ4ETiulOr1h5dhhBrukwoqM3GOG97Lr+jzgpsxcBpwFfD8i9vjaEXFhRGyJiC1TU1P7Xm3DrNu8k49cfx/bdr1UdSmSRkyRcJ8Ejuy4Xsae0y7nA7cBZOZ9wEJgUfcXysy1mTmRmROLFy+eW8UN0rmNrZtiSdoXRaZlHgBWRMQY8BRwLvDRrjY7gfcCN0XEsbTC3aF5CdzGVtJc9By5Z+arwCXAncCjtFbFbI2IqyLi7HazzwMXRMSvgJuBT2Rm99SNJGlACq1zb69Z39j13pUdr7cBJ5dbWrOt27yTzY8/X+gJUknq5hOqQ6Z7h0Xn2iXNheE+ZKZvovqUqaT5MNyHSOdUjDdRJc1HkaWQGoDOLQacipE0X4b7kHCLAUllMtyHiFsMSCqL4T4EpufaJakshvsQmJ6Sca5dUlkM94p1rpBxSkZSWQz3CrlCRlK/GO4VcoWMpH4x3CvmdIykfjDcJamGDHdJqiHDXZJqyHCXpBoy3CviU6mS+slwr4hPpUrqJ8O9Qi6DlNQvhrsk1ZDhLkk1ZLhXwJupkvrNcK+AN1Ml9ZsHZA/Qus07Wf/QU2zb9ZI3UyX1lSP3AZoO9vElBztql9RXjtwHbHzJwdx60UlVlyGp5hy5S1INGe6SVEOGuyTVkOEuSTVkuA+IDy5JGqRC4R4RZ0TE9ojYERGXzdLmwxGxLSK2RsS6csscfT64JGmQei6FjIgFwHXA+4BJ4IGI2JCZ2zrarAD+Gjg5M1+IiCP6VfAo88ElSYNSZOR+IrAjMx/LzJeBW4A1XW0uAK7LzBcAMvOZcsscbU7JSBq0IuG+FHiy43qy/V6nY4BjIuLeiLg/Is6Y6QtFxIURsSUitkxNTc2t4hHklIykQSsS7jHDe9l1vT+wAjgVOA/4TkQcssdvylybmROZObF48eJ9rXWkOSUjaZCKhPskcGTH9TLg6RnarM/MVzLzcWA7rbCXJFWgSLg/AKyIiLGIeBtwLrChq80dwGqAiFhEa5rmsTILlSQV1zPcM/NV4BLgTuBR4LbM3BoRV0XE2e1mdwLPRcQ24G7grzLzuX4VLUnau0K7QmbmRmBj13tXdrxO4HPtX5KkivmEap+5DFJSFQz3PnMZpKQqGO4D4DJISYNmuPeRUzKSqmK495FTMpKqYrj3mVMykqpguEtSDRnuklRDhrsk1ZDhLkk1ZLj3icsgJVXJcO8Tl0FKqpLh3gfTo3aXQUqqiuHeB47aJVXNcC+Zo3ZJw8BwL5mjdknDwHDvA0ftkqpmuEtSDRnuklRDhrsk1ZDhLkk1ZLiXyC0HJA0Lw71ELoOUNCwM95K5DFLSMDDcJamG9q+6gDpYt3kn6x96im27XmJ8ycFVlyNJjtzL0BnszrdLGgaO3Oepc6OwWy86qepyJAlw5D5vrpCRNIwM93lwe19Jw6pQuEfEGRGxPSJ2RMRle2n3oYjIiJgor8Th5ahd0rDqGe4RsQC4DjgTGAfOi4jxGdodBHwa2Fx2kcPMUbukYVRk5H4isCMzH8vMl4FbgDUztPsK8DVgd4n1DS23GpA0zIqE+1LgyY7ryfZ7b4iIdwFHZuaPS6xtaK3bvJPLb38EcEpG0nAqEu4xw3v5xocR+wHfAD7f8wtFXBgRWyJiy9TUVPEqh8z0XPvV5xzvlIykoVQk3CeBIzuulwFPd1wfBBwH3BMRTwCrgA0z3VTNzLWZOZGZE4sXL5571RVZt3knH7n+Prbtesm5dklDrchDTA8AKyJiDHgKOBf46PSHmfkisGj6OiLuAf4yM7eUW2q1OqdiVo4d5nSMpKHWM9wz89WIuAS4E1gA3JCZWyPiKmBLZm7od5HDwKkYSaOk0PYDmbkR2Nj13pWztD11/mUNj85NwZyKkTQqfEK1BzcFkzSK3DhsFt3b+LopmKRR4sh9Fo7YJY0yR+574Yhd0qhy5C5JNWS4S1INGe4zcFMwSaPOcO/ipmCS6sBw79AZ7D6JKmmUGe5tBrukOjHc29w7RlKdGO540LWk+ml8uHsDVVIdNT7cnY6RVEeND3fA6RhJtWO4S1INGe6SVEOGuyTVkOEuSTXU6HB3gzBJddXIwzqmj9CbDnbXt0uqm8aFe+dDSyvHDmPNCUtdBimpdhoX7j60JKkJGjnn7kNLkuquUeHuDVRJTdGYcHeDMElN0phwd65dUpM0JtzBuXZJzdGocJekpmhEuHsjVVLT1D7cvZEqqYkKhXtEnBER2yNiR0RcNsPnn4uIbRHxcETcFRFHl1/q3HgjVVIT9Qz3iFgAXAecCYwD50XEeFezXwITmfnHwA+Br5Vd6Hx4I1VS0xQZuZ8I7MjMxzLzZeAWYE1ng8y8OzN/2768H1hWbpmSpH1RJNyXAk92XE+235vN+cBPZvogIi6MiC0RsWVqaqp4lZKkfVIk3GOG93LGhhEfByaAr8/0eWauzcyJzJxYvHhx8SolSfukyK6Qk8CRHdfLgKe7G0XEacAXgPdk5u/KKU+SNBdFRu4PACsiYiwi3gacC2zobBAR7wKuB87OzGfKL1OStC96hntmvgpcAtwJPArclplbI+KqiDi73ezrwB8AP4iIhyJiwyxfTpI0AIUO68jMjcDGrveu7Hh9Wsl1SZLmofZPqEpSExnuklRDtQ53NwyT1FS1DvfpfWXcMExS09Q63MF9ZSQ1U23D3SkZSU1W23B3SkZSk9U23MEpGUnNVetwl6SmMtwlqYZqGe7eTJXUdLUMd2+mSmq62oX79Kjdm6mSmqx24e6oXZIKbvk77NZt3vlGqG/b9ZKjdkmNV4uR+/qHnmLbrpcAGF9ysKN2SY1Xi5E7tEL91otOqroMSRoKtRi5S5LeynCXpBoy3CWphgx3Saohw12Sashwl6QaGvlwd5MwSdrTyIe72w1I0p5GPtzBE5ckqVstwl2S9FaGuyTVkOEuSTU00uHuShlJmlmhcI+IMyJie0TsiIjLZvj89yLi1vbnmyNiedmFzsSVMpI0s57hHhELgOuAM4Fx4LyIGO9qdj7wQmb+EfAN4JqyC+3mcXqSNLsiI/cTgR2Z+VhmvgzcAqzparMG+G779Q+B90ZElFfmW63bvJPLb3+k9Y0dtUvSHooc1rEUeLLjehJYOVubzHw1Il4EDgeeLaPITl/+0VZuvPcJAK4+53hH7ZI0gyLhPtMIPOfQhoi4ELgQ4Kij5h7KK8cOY80JSw12SZpFkXCfBI7suF4GPD1Lm8mI2B94O7DHMpbMXAusBZiYmNgj/Iv44gfeOZffJkmNUmTO/QFgRUSMRcTbgHOBDV1tNgB/1n79IeDnmTmn8JYkzV/PkXt7Dv0S4E5gAXBDZm6NiKuALZm5AfgH4PsRsYPWiP3cfhYtSdq7ItMyZOZGYGPXe1d2vN4N/Em5pUmS5mqkn1CVJM3McJekGjLcJamGDHdJqiHDXZJqKKpajh4RU8Bv5vjbF9GHrQ2GnH1uBvvcDPPp89GZubhXo8rCfT4iYktmTlRdxyDZ52awz80wiD47LSNJNWS4S1INjWq4r626gArY52awz83Q9z6P5Jy7JGnvRnXkLknai6EO92E9mLufCvT5cxGxLSIejoi7IuLoKuosU68+d7T7UERkRIz8yooifY6ID7d/1lsjYt2gayxbgT/bR0XE3RHxy/af77OqqLMsEXFDRDwTEb+e5fOIiG+2/3s8HBHvLrWAzBzKX7S2F/5P4B3A24BfAeNdbf4C+Hb79bnArVXXPYA+rwZ+v/36U03oc7vdQcAm4H5gouq6B/BzXgH8Eji0fX1E1XUPoM9rgU+1X48DT1Rd9zz7fArwbuDXs3x+FvATWifZrQI2l/n9h3nkPnQHcw9Azz5n5t2Z+dv25f20TsYaZUV+zgBfAb4G7B5kcX1SpM8XANdl5gsAmfnMgGssW5E+J3Bw+/Xb2fPEt5GSmZuY4US6DmuA72XL/cAhEbGkrO8/zOE+08HcS2drk5mvAtMHc4+qIn3udD6t//OPsp59joh3AUdm5o8HWVgfFfk5HwMcExH3RsT9EXHGwKrrjyJ9/hLw8YiYpHV+xKWDKa0y+/r3fZ8UOqyjIqUdzD1CCvcnIj4OTADv6WtF/bfXPkfEfsA3gE8MqqABKPJz3p/W1MyptP519q8RcVxm/nefa+uXIn0+D7gpM/82Ik6idbrbcZn5ev/Lq0Rf82uYR+77cjA3ezuYe4QU6TMRcRrwBeDszPzdgGrrl159Pgg4DrgnIp6gNTe5YcRvqhb9s70+M1/JzMeB7bTCflQV6fP5wG0AmXkfsJDWHix1Vejv+1wNc7g38WDunn1uT1FcTyvYR30eFnr0OTNfzMxFmbk8M5fTus9wdmZuqabcUhT5s30HrZvnRMQiWtM0jw20ynIV6fNO4L0AEXEsrXCfGmiVg7UB+NP2qplVwIuZuau0r171HeUed5vPAv6d1l32L7Tfu4rWX25o/fB/AOwAfgG8o+qaB9DnfwH+C3io/WtD1TX3u89dbe9hxFfLFPw5B/B3wDbgEeDcqmseQJ/HgXtpraR5CDi96prn2d+bgV3AK7RG6ecDnwQ+2fEzvq793+ORsv9c+4SqJNXQME/LSJLmyHCXpBoy3CWphgx3Saohw12SashwV6NFxKcj4tGI+Meqa5HK5FJINVpE/BtwZraeAu3VdkFmvjaAsqR5c+SuxoqIb9PagnZDRLwYEd+PiJ9HxH9ExAXtNqe29xhfR+tBE2kkOHJXo7X3q5kALgHOobV3zYG09lJfSeux/38CjisyupeGhSN36U3rM/P/MvNZ4G5ae5AD/MJg16gx3KU3df8zdvr6fwddiDRfhrv0pjURsTAiDqe1j/oDFdcjzZnhLr3pF7Tm1+8HvpKZI33Mm5rNG6oSEBFfAv4nM6+tuhapDI7cJamGHLlLUg05cpekGjLcJamGDHdJqiHDXZJqyHCXpBoy3CWphv4foQ30oYWh+IEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.7 预测值靠近1或0的ROC曲线\n",
    "阈值为0.5时，我们让预测值有80%的可能性是正确预测，但值靠近1或0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(50)\n",
    "\n",
    "def f(x):\n",
    "    if x == 1:\n",
    "        if rand() > 0.2:\n",
    "            return 1 - rand() / 10\n",
    "        else:\n",
    "            return rand() / 10\n",
    "    else:\n",
    "        if rand() > 0.2:\n",
    "            return rand() / 10\n",
    "        else:\n",
    "            return 1 - rand() / 10\n",
    "        \n",
    "y_pred = np.array([f(yi) for yi in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。可以看出曲线跟10.6几乎没有差异，根据ROC曲线的定义不难思考出原因。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.812.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xa09e4a8>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFI1JREFUeJzt3X+s3XV9x/HnmwJrxqj8aElqf9CalYQrJmpuaBsS1kYkBSMNxrkyzeZCKDpAjWYJQ4IOEx2KMzNp1Mb5Mys/XCK9c1XmENPF0NoSEexl3TrAcqEZpTDIxiq/3vvjnAvH23Pv+fbe7znfc77n+UianO85n3vO+8O5ffHp5/v5fr6RmUiS6uWEqguQJJXPcJekGjLcJamGDHdJqiHDXZJqyHCXpBoy3CWphgx3Saohw12SaujEqj544cKFuWLFiqo+XpIG0v333/90Zi7q1K6ycF+xYgV79+6t6uMlaSBFxK+LtHNaRpJqyHCXpBoy3CWphiqbc2/npZdeYmJigqNHj1ZdypzNnz+fpUuXctJJJ1VdiqQh1FfhPjExwamnnsqKFSuIiKrLmbXM5MiRI0xMTLBy5cqqy5E0hDpOy0TENyLiqYj41TSvR0R8OSIORMSDEfH22RZz9OhRzjzzzIEOdoCI4Mwzz6zFv0AkDaYic+7fAjbM8PolwKrmn83AV+ZS0KAH+6S69EPSYOo4LZOZOyNixQxNNgLfycb9+nZFxGkRsTgzD5VUoyQNvG27D7L9gScAGHnjAj717jd39fPKmHNfAjzecjzRfO6YcI+IzTRG9yxfvryEj5ak/tMa5JN2P/oMAKtXntGTGsoI93bzD23vup2ZW4GtAKOjo31/Z+7MJDM54QRXjEo6VrsQh/ZBvnrlGWx86xL+eHVvBrZlhPsEsKzleCnwZAnvW4nHHnuMSy65hPXr13PffffxsY99jFtvvZXM5F3vehe33HILAD/60Y+44YYbeOWVV1i4cCH33HNPxZVLKst0oT3VdKPxXgd5O2WE+xhwbUTcDqwGnitjvv2v/nEf408+P+fiWhWd59q/fz/f/OY3ufHGG1mzZg33338/p59+OhdffDF33XUXF1xwAVdddRU7d+5k5cqVPPPMM6XWKala2x94gvFDzzOyeMGM7fohxKfTMdwj4jZgHbAwIiaATwEnAWTmV4EdwKXAAeAF4M+6VWyvnH322axZs4bt27ezbt06Fi1qbMD2/ve/n507dzJv3jwuvPDC19awn3FGb+bQJJWv3Sh9MtjvuHptRVXNXZHVMld0eD2Ba0qrqKnbZ5JncsoppwCNOfd2MtOljlJNtBuljyxewMa3LqmwqrnrqytU+83q1av56Ec/ytNPP83pp5/ObbfdxnXXXcfatWu55pprePTRR1+blnH0LvW36ebR6zBKb8dwn8HixYv53Oc+x/r168lMLr30UjZu3AjA1q1bec973sOrr77KWWedxY9//OOKq5U0nW27D3LD9x8Cjj35WYdRejsx3dRDt42OjubUm3U8/PDDnHvuuZXU0w1164/UD4quZGk1uarls5e/pS9Pfh6PiLg/M0c7tXPkLqkvzHX54Uz6eVVLtxjukrrmeEbZRUN7GIN6Nvou3OuyEqWq6S6p12YK8OMZZRva5eqrcJ8/fz5HjhwZ+G1/J/dznz9/ftWlSF0zGeozBbiBXZ2+CvelS5cyMTHB4cOHqy5lzibvxCT1q9mcmGzVGuoGeP/pq3A/6aSTvHOR1EWtgT7XXQoN9f7WV+Eu6fjN9qSl4Vxvhrs0YKaGuSct1Y7hLvWh41mBYmCrHcNd6hNF58MNcxVhuEs9VuTuPQa45spwl3qgyKjcQFeZDHepBJ1WrDgqV68Z7tIsHc+acQNdvWa4S7MwdX9ww1v9xnCXZtDp5Gcd9gdXPRnuGjplbEPrSF39znDX0Ciyi+FUhrgGleGu2prpMn0DW3VnuKt2phuhG+oaJoa7amf7A08wfuh5w1xDzXBXJeZ6o4iZjB96npHFC7jj6rVdeX9pEBju6qnZnNQ8XiOLF7DxrUtKf19pkBju6pp2o3NPakq9YbirK6ZewTnJUJd6w3DXa8qcB/cKTqlahvuQKBLcZc6DO0KXqlUo3CNiA/C3wDzg65n511NeXw58Gzit2eb6zNxRcq0qoMiNIKZjIEv10THcI2IesAV4JzAB7ImIscwcb2l2I3BnZn4lIkaAHcCKLtQ7dI53qsS9UCRBsZH7+cCBzHwEICJuBzYCreGewILm4zcAT5ZZ5DCYy4i7lSEuCYqF+xLg8ZbjCWD1lDafBv45Iq4DTgEuKqW6ITHdypLJY8Na0vEqEu7R5rmccnwF8K3M/GJErAW+GxHnZearv/VGEZuBzQDLlxtWkyZH7K4skVSWEwq0mQCWtRwv5dhplyuBOwEy8z5gPrBw6htl5tbMHM3M0UWLFs2u4ppavfIMg11SaYqE+x5gVUSsjIiTgU3A2JQ2B4F3AETEuTTC/XCZhUqSiusY7pn5MnAtcDfwMI1VMfsi4uaIuKzZ7BPAVRHxS+A24IOZOXXqRpLUI4XWuTfXrO+Y8txNLY/HgQvKLW04bNt9kN2PPtOVDbQkDa8i0zLqktZVMu5iKKlMhnuFXCUjqVsM94q0TscY7JLKZrhXZHLU7nSMpG4w3CvkqF1StxjuklRDhrsk1ZA36+ixyd0fxw89z8jiBZ1/QJJmwZF7j7UGuydTJXWLI/cKjCxewB1Xr626DEk15si9hybXtktStxnuPeJWA5J6yXDvEbcakNRLhnsPedGSpF4x3CWphgz3HvBEqqReM9x7wE3CJPWa4d4jzrdL6iXDXZJqyHDvMufbJVXBcO8y59slVcFw7wHn2yX1muEuSTVkuHeR8+2SqmK4d5Hz7ZKqYrh3yeSo3fl2SVUw3LvEUbukKnknphJN3h8VYPzQ847aJVXGkXuJJu+PCniPVEmVcuReMu+PKqkfOHKXpBoqFO4RsSEi9kfEgYi4fpo274uI8YjYFxHbyi2z/7mmXVI/6TgtExHzgC3AO4EJYE9EjGXmeEubVcBfAhdk5rMRcVa3Cu5H3vxaUr8pMnI/HziQmY9k5ovA7cDGKW2uArZk5rMAmflUuWX2r9Zg9+bXkvpFkXBfAjzecjzRfK7VOcA5EfGziNgVERvavVFEbI6IvRGx9/Dhw7OruM9MLn002CX1kyLhHm2eyynHJwKrgHXAFcDXI+K0Y34oc2tmjmbm6KJFi4631r7lenZJ/aZIuE8Ay1qOlwJPtmmzPTNfysxHgf00wr7WPIkqqV8VCfc9wKqIWBkRJwObgLEpbe4C1gNExEIa0zSPlFloP3KLAUn9qmO4Z+bLwLXA3cDDwJ2ZuS8ibo6Iy5rN7gaORMQ4cC/wF5l5pFtF9xOnZCT1o0JXqGbmDmDHlOduanmcwMebfyRJFfMKVUmqIcN9ljyZKqmfGe6z5MlUSf3McJ8DT6ZK6leGuyTVkOEuSTVkuEtSDRnuklRDhrsk1ZDhLkk1ZLhLUg0Z7rPg1amS+p3hPgtenSqp3xXaFVIN23YfZPsDTzB+6HmvTpXU1xy5H4fJYB9ZvMBRu6S+5sj9OI0sXsAdV6+tugxJmpEjd0mqIUfuBbTOtY8sXlB1OZLUkSP3ApxrlzRoHLkX5Fy7pEHiyL0DL1iSNIgM9w68YEnSIDLcC/CCJUmDxnCXpBoy3CWphgx3Saohl0JOwwuXJA0yR+7T8MIlSYPMkfsMvHBJ0qBy5C5JNVQo3CNiQ0Tsj4gDEXH9DO3eGxEZEaPlldh7XpUqadB1DPeImAdsAS4BRoArImKkTbtTgY8Au8suste8KlXSoCsycj8fOJCZj2Tmi8DtwMY27T4DfB44WmJ9lfGqVEmDrEi4LwEebzmeaD73moh4G7AsM39QYm2SpFkqEu7R5rl87cWIE4AvAZ/o+EYRmyNib0TsPXz4cPEqJUnHpUi4TwDLWo6XAk+2HJ8KnAf8NCIeA9YAY+1Oqmbm1swczczRRYsWzb5qSdKMioT7HmBVRKyMiJOBTcDY5IuZ+VxmLszMFZm5AtgFXJaZe7tSsSSpo47hnpkvA9cCdwMPA3dm5r6IuDkiLut2gb3mMkhJdVDoCtXM3AHsmPLcTdO0XTf3sqqxbfdBbvj+Q4DLICUNNq9QbTG5vv2zl7/FZZCSBprhPoXr2yXVgeEuSTVkuEtSDRnuklRDhrsk1ZDh3uT6dkl1Yrg3uc2vpDox3Fu4DFJSXRjuklRDhrsk1ZDhjidTJdXP0Ie7m4VJqqOhD3c3C5NUR0Mf7uAqGUn1Y7hLUg0Ndbh7IlVSXQ11uHtVqqS6GupwB+fbJdXT0Ie7JNWR4S5JNWS4S1INDW24u1JGUp0NZbi75YCkuhvKcHfLAUl1N5ThDi6BlFRvQxvuklRnQxfunkiVNAyGLtzdckDSMBi6cAfn2yXV34lVF9Ar23YfZPsDTzB+6HlGFi+ouhxJ6qpCI/eI2BAR+yPiQERc3+b1j0fEeEQ8GBH3RMTZ5Zc6N63B7pSMpLrrOHKPiHnAFuCdwASwJyLGMnO8pdkvgNHMfCEiPgx8HvijbhQ8FyOLF3DH1WurLkOSuq7IyP184EBmPpKZLwK3AxtbG2TmvZn5QvNwF7C03DLnxhUykoZNkXBfAjzecjzRfG46VwI/bPdCRGyOiL0Rsffw4cPFq5wjV8hIGjZFwj3aPJdtG0Z8ABgFvtDu9czcmpmjmTm6aNGi4lWWwBUykoZJkXCfAJa1HC8FnpzaKCIuAj4JXJaZvymnvLlzSkbSMCoS7nuAVRGxMiJOBjYBY60NIuJtwNdoBPtT5Zc5O+7+KGlYdQz3zHwZuBa4G3gYuDMz90XEzRFxWbPZF4DfA74XEQ9ExNg0b9dT7v4oaVgVuogpM3cAO6Y8d1PL44tKrmvOJqdjnGuXNIxquf2A0zGShl3twr012J2OkTSsahfuzrNLUo02DmvdGMx5dknDrjYjdzcGk6TX1WLk3royxo3BJKkmI3f3jpGk31aLcAf3jpGkVrUJd0nS6wx3Saohw12Sashwl6QaMtwlqYYGPty9GYckHWvgw9017pJ0rIEPd3CNuyRNVYtwlyT9NsNdkmpooMPdk6mS1N5Ah7snUyWpvYEOd/BkqiS1M/DhLkk6luEuSTVkuEtSDRnuklRDhrsk1ZDhLkk1NLDh7gVMkjS9gQ13L2CSpOkNbLiDFzBJ0nQKhXtEbIiI/RFxICKub/P670TEHc3Xd0fEirILlSQV1zHcI2IesAW4BBgBroiIkSnNrgSezczfB74E3FJ2oa2cb5ekmRUZuZ8PHMjMRzLzReB2YOOUNhuBbzcf/wPwjoiI8sp83bbdB7nh+w81PtT5dklqq0i4LwEebzmeaD7Xtk1mvgw8B5xZRoFTTZ5I/ezlb3G+XZKmcWKBNu1G4DmLNkTEZmAzwPLlswvmkTcuYOSNCwx2SZpBkXCfAJa1HC8FnpymzUREnAi8AThmUjwztwJbAUZHR48J/yI+9e43z+bHJGmoFJmW2QOsioiVEXEysAkYm9JmDPjT5uP3Aj/JzFmFtyRp7jqO3DPz5Yi4FrgbmAd8IzP3RcTNwN7MHAP+DvhuRBygMWLf1M2iJUkzKzItQ2buAHZMee6mlsdHgT8stzRJ0mwN9BWqkqT2DHdJqiHDXZJqyHCXpBoy3CWphqKq5egRcRj49Sx/fCHwdInlDAL7PBzs83CYS5/PzsxFnRpVFu5zERF7M3O06jp6yT4PB/s8HHrRZ6dlJKmGDHdJqqFBDfetVRdQAfs8HOzzcOh6nwdyzl2SNLNBHblLkmbQ1+E+jDfmLtDnj0fEeEQ8GBH3RMTZVdRZpk59bmn33ojIiBj4lRVF+hwR72t+1/siYluvayxbgd/t5RFxb0T8ovn7fWkVdZYlIr4REU9FxK+meT0i4svN/x4PRsTbSy0gM/vyD43thf8TeBNwMvBLYGRKmz8Hvtp8vAm4o+q6e9Dn9cDvNh9/eBj63Gx3KrAT2AWMVl13D77nVcAvgNObx2dVXXcP+rwV+HDz8QjwWNV1z7HPFwJvB341zeuXAj+kcSe7NcDuMj+/n0fufXVj7h7p2OfMvDczX2ge7qJxZ6xBVuR7BvgM8HngaC+L65Iifb4K2JKZzwJk5lM9rrFsRfqcwILm4zdw7B3fBkpm7qTNHelabAS+kw27gNMiYnFZn9/P4d5XN+bukSJ9bnUljf/zD7KOfY6ItwHLMvMHvSysi4p8z+cA50TEzyJiV0Rs6Fl13VGkz58GPhAREzTuH3Fdb0qrzPH+fT8uhW7WUZHSbsw9QAr3JyI+AIwCf9DVirpvxj5HxAnAl4AP9qqgHijyPZ9IY2pmHY1/nf1rRJyXmf/d5dq6pUifrwC+lZlfjIi1NO7udl5mvtr98irR1fzq55H78dyYm5luzD1AivSZiLgI+CRwWWb+pke1dUunPp8KnAf8NCIeozE3OTbgJ1WL/m5vz8yXMvNRYD+NsB9URfp8JXAnQGbeB8ynsQdLXRX6+z5b/Rzuw3hj7o59bk5RfI1GsA/6PCx06HNmPpeZCzNzRWauoHGe4bLM3FtNuaUo8rt9F42T50TEQhrTNI/0tMpyFenzQeAdABFxLo1wP9zTKntrDPiT5qqZNcBzmXmotHev+oxyh7PNlwL/TuMs+yebz91M4y83NL787wEHgJ8Db6q65h70+V+A/wIeaP4Zq7rmbvd5StufMuCrZQp+zwH8DTAOPARsqrrmHvR5BPgZjZU0DwAXV13zHPt7G3AIeInGKP1K4EPAh1q+4y3N/x4Plf177RWqklRD/TwtI0maJcNdkmrIcJekGjLcJamGDHdJqiHDXUMtIj4SEQ9HxN9XXYtUJpdCaqhFxL8Bl2TjKtBObedl5is9KEuaM0fuGloR8VUaW9CORcRzEfHdiPhJRPxHRFzVbLOuucf4NhoXmkgDwZG7hlpzv5pR4Frgchp715xCYy/11TQu+/8n4Lwio3upXzhyl163PTP/LzOfBu6lsQc5wM8Ndg0aw1163dR/xk4e/2+vC5HmynCXXrcxIuZHxJk09lHfU3E90qwZ7tLrfk5jfn0X8JnMHOjbvGm4eUJVAiLi08D/ZOatVdcilcGRuyTVkCN3SaohR+6SVEOGuyTVkOEuSTVkuEtSDRnuklRDhrsk1dD/A8xC3wiKOr3jAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.8 预测值恒等于0.9的ROC曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.array([0.9] * len(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算出ROC曲线的数据点，并绘制ROC曲线。发现预测值为定值的模型，ROC曲线是一条直线，AUC恒等于0.5。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC is 0.500.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xa0ff0f0>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XlYlXX+//HnR0RRxBVxAwR3EbQMRbNRyyyzxdSabN+t5tt35jdLaqll2WI1TdN8sxprWqddTMlM20zbLLWSTVHEBVwBFRd2zuf3B9QwRHLUAzfnnNfjuryuczgfz3ndgi9v73Of922stYiIiG9p4nQAERHxPJW7iIgPUrmLiPgglbuIiA9SuYuI+CCVu4iID1K5i4j4IJW7iIgPUrmLiPigpk69cGhoqI2KinLq5UVEvNL69evzrLUd61rnWLlHRUWxbt06p15eRMQrGWN2uLNOh2VERHyQyl1ExAep3EVEfJBjx9xrU1ZWRk5ODsXFxU5HOWVBQUGEh4cTGBjodBQR8UONqtxzcnIICQkhKioKY4zTcU6atZb8/HxycnKIjo52Oo6I+KE6D8sYY140xuw3xqT+yuPGGPMPY0ymMSbZGDP4ZMMUFxfToUMHry52AGMMHTp08In/gYiId3LnmPvLwLjjPH4B0Lvq11Tg2VMJ5O3F/hNf2Q4R8U51lru1djVw4DhLJgCv2kprgLbGmC6eCigi4iuKSit45MON5BwsrPfX8sTZMt2A7Gr3c6q+9gvGmKnGmHXGmHW5ubkeeGkREe/w9dY8zv/7av65KouVGfXff554Q7W24w+1XnXbWrsAWAAQHx/f6K/Mba3FWkuTJjpjVEROzuHiMh5ZtpE3v8smqkNL3po6jGE9OtT763qitXKAiGr3w4HdHnheR2zfvp3+/fvzu9/9jsGDB/Paa68RFxdHbGws06dP/3nd8uXLGTx4MIMGDWLMmDEOJhaRxurj9H2M/dsq3l6bzW2jerD8/41skGIHz+y5JwF3GmPeAhKAAmvtnlN90vvfTyN99+FTDlddTNfW3HfxgDrXZWRk8NJLLzFr1iyGDRvG+vXradeuHeeddx6LFy9mxIgR3HrrraxevZro6GgOHDjeWxIi4m/yjpYwJymNpcl76Nc5hOevi2dgeNsGzVBnuRtj3gRGA6HGmBzgPiAQwFr7HLAMGA9kAoXAjfUVtqF0796dYcOGsWTJEkaPHk3HjpUD2K6++mpWr15NQEAAI0eO/Pkc9vbt2zsZV0QaCWstS37czf3vp3GspII/j+3DbaN60qxpwx/arbPcrbVX1vG4Bf7HY4mquLOHXV+Cg4OBym9Ubay1OtVRRP7L7kNFzFqcymeb9nN6ZFsemzyQ3p1CHMujdwqPIyEhgVWrVpGXl0dFRQVvvvkmo0aNYvjw4axatYpt27YB6LCMiB9zuSz/XrOD855czTdb87n3ohgW3n6mo8UOjWz8QGPTpUsXHnnkEc4++2ystYwfP54JEyYAsGDBAiZNmoTL5SIsLIyPP/7Y4bQi0tC25R1jemIy3207wFm9QnlkUhwR7Vs6HQsA82uHHupbfHy8rXmxjo0bN9K/f39H8tQHX9seEalUXuHihS+38eTHm2nWtAmzL4zh8vjwBjlca4xZb62Nr2ud9txFRE5A+u7DTE9MJmVXAefFdGLupbF0ah3kdKxfULmLiLihpLyCpz/L5NnPt9K2ZSDzrxrM+LjOjfbkikZX7r5yJopTh7tExPPW7zjI9MRkMvcfZdLgbsy+MIZ2wc2cjnVcjarcg4KCyM/P9/qxvz/Ncw8Kanz/VRMR9xWWlvP4igxe/no7Xdu04OUbhzC6b5jTsdzSqMo9PDycnJwcfGGo2E9XYhIR7/TlljxmLEom52AR1w3vzrRx/WjVvFFV5nE1qqSBgYG6cpGIOKqgsIyHlqXzzroceoQG885twxka7X2fQm9U5S4i4qTlqXuZvSSVA8dKuWN0T/4wpjdBgQFOxzopKncR8Xu5RyoHfX2QsoeYLq156YYhxHZr43SsU6JyFxG/Za1l0fe7eGBpOkWlFdx1fl+mjuxBYID3T2ZRuYuIX9p1qIh7FqWwanMuZ3Rvx6OTB9IrrJXTsTxG5S4ifsXlsvz72x08+uEmLHD/JQO4dlh3mjTx3tOva6NyFxG/sTX3KDMSk1m7/SC/6R3KwxMbz6AvT1O5i4jPK6tw8fwXWfz9ky20CAzgr5cPYvLgbl79Ycm6qNxFxKel7ipgemIyabsPc0FsZ+6fMICwEN//9LjKXUR8UnFZBf/32RaeW5VFu5bNePbqwVwQ18XpWA1G5S4iPmfd9gNMS0wmK/cYl58RzswL+9O2ZeMe9OVpKncR8RlHS8p5fPkmXl2zg65tWvDqTUMZ2aej07EcoXIXEZ+wanMu9yxKYXdBEdcPj+Ku8/sS7EWDvjzNf7dcRHzCocJS5i7dSOL3OfTsGMy7tw0nPsr7Bn15mspdRLzWhyl7mL0kjYOFpdx5di/uPKeX1w768jSVu4h4nf2Hi7l3SRrL0/YS2601r9w0hAFdvXvQl6ep3EXEa1hrWbg+h7lL0ykudzF9XD9u/U00TX1g0JenqdxFxCtkHyjknvdS+GJLHkOj2jNvchw9OvrOoC9PU7mLSKNW4bK8+s12Hl+RgQHmThjA1Qm+N+jL01TuItJoZe4/wvTEFNbvOMioPh15eFIc3dq2cDqWV1C5i0ijU1bh4p+rtvKPTzNp2TyAv/12EBNP9+1BX56mcheRRiUlp4Bpicls3HOYCwd2Yc7FA+gY0tzpWF7HrXI3xowDngICgBestfNqPB4JvAK0rVozw1q7zMNZRcSHFZdV8PdPtvD8F1l0CG7GP689g/MHdHY6lteqs9yNMQHAfGAskAOsNcYkWWvTqy2bBbxjrX3WGBMDLAOi6iGviPigb7PymbEohW15x7giPoJ7LuxPmxaBTsfyau7suQ8FMq21WQDGmLeACUD1crdA66rbbYDdngwpIr7pSHEZjy3P4LU1O4ho34LXb0lgRK9Qp2P5BHfKvRuQXe1+DpBQY80c4CNjzP8CwcC5HkknIj5rZcZ+Zi5KYc/hYm4aEc1fzu9Dy2Z6G9BT3PmTrO3taVvj/pXAy9baJ4wxw4HXjDGx1lrXfz2RMVOBqQCRkZEnk1dEvNzBY6XMXZrOoh920TusFYl3nMngyHZOx/I57pR7DhBR7X44vzzscjMwDsBa+40xJggIBfZXX2StXQAsAIiPj6/5D4SI+DBrLR+k7OG+JWkUFJXx+zG9+Z+ze9K8qQZ91Qd3yn0t0NsYEw3sAqYAV9VYsxMYA7xsjOkPBAG5ngwqIt5r3+FiZi1O5eP0fQwMb8O/b0mgf5fWdf9GOWl1lru1ttwYcyewgsrTHF+01qYZYx4A1llrk4A/A88bY/5I5SGbG6y12jMX8XPWWt5Zl82DH2yktNzFPeP7cdMIDfpqCG69e1F1zvqyGl+7t9rtdGCEZ6OJiDfbmV/IjEXJfL01n4To9jw6eSBRocFOx/IbemtaRDyqwmV5+evt/HVFBgFNDA9NjOXKIZEa9NXAVO4i4jGb9x1h2sJkfsw+xDn9wnhoYixd2mjQlxNU7iJyykrLXTz7+VaeXrmFVs2b8tSU07hkUFcN+nKQyl1ETsmG7ENMT0xm094jXDKoK/ddHEOHVhr05TSVu4iclKLSCp78ZDMvfJFFWEgQL1wXz7kxnZyOJVVU7iJywr7Zms/di5LZnl/IlUMjuXt8P1oHadBXY6JyFxG3HS4uY96Hm3jj251079CSN25N4MyeGvTVGKncRcQtn27cx8z3Utl/pJhbfxPNn8b2pUUzjQ5orFTuInJc+UdLuP/9dJI27KZvpxCeu/YMToto63QsqYPKXURqZa0lacNu7n8/nSPFZfzx3D7cMbonzZpqdIA3ULmLyC/sKShi1nupfLppP4Mi2vLY5IH07RzidCw5ASp3EfmZy2V5a202jyzbSJnLxawL+3PjiGgCNDrA66jcRQSA7XnHmLEomTVZBxjeowPzJsfRvYMGfXkrlbuInyuvcPHSV9t54uMMAps0Yd6kOK4YEqHRAV5O5S7ixzbtPcz0hclsyCng3P6dePDSWDq3CXI6lniAyl3ED5WUVzB/5VaeWZlJmxaB/N+Vp3PRwC7aW/chKncRP/PDzoNMT0xm876jTDy9G7MviqF9cDOnY4mHqdxF/ERhaTlPfLSZF7/aRufWQbx4Qzzn9NOgL1+lchfxA19n5jFjUQo7DxRyzbBIpo/rR4gGffk0lbuIDysoKuORZRt5a2020aHBvDV1GMN6dHA6ljQAlbuIj/oobS+zFqeSd7SE20b14I/n9iEoUIO+/IXKXcTH5B0tYU5SGkuT99CvcwgvXB/PwHAN+vI3KncRH2GtZfGPu7j//XQKSyr489g+3D66J4EBGvTlj1TuIj5g96EiZr6XwsqMXE6PrBz01buTBn35M5W7iBdzuSyvf7eTRz/cRIXLcu9FMVx/ZpQGfYnKXcRbZeUeZUZiCt9tP8BZvUJ5ZFIcEe1bOh1LGgmVu4iXKa9w8cKX23jy4800b9qExy4byOVnhGt0gPwXlbuIF0nffZhpiRtI3XWY8wd0Yu6EWMJaa9CX/JLKXcQLlJRX8PRnmTz7+VbatgzkmasHc0FsZ+2ty69SuYs0cut3HGB6YgqZ+48yaXA3Zl8YQzsN+pI6qNxFGqljJeU8viKDV77ZTtc2LXj5xiGM7hvmdCzxEm6VuzFmHPAUEAC8YK2dV8ua3wJzAAtssNZe5cGcIn7liy253L0ohZyDRVw/vDt3jetHq+baFxP31fnTYowJAOYDY4EcYK0xJslam15tTW/gbmCEtfagMUa7FyInoaCwjAc/SOfd9Tn06BjMu7cPZ0hUe6djiRdyZ1dgKJBprc0CMMa8BUwA0qutuRWYb609CGCt3e/poCK+bnnqXmYvSeXAsVJ+N7onvx/TW4O+5KS5U+7dgOxq93OAhBpr+gAYY76i8tDNHGvt8ppPZIyZCkwFiIyMPJm8Ij5n/5Fi5iSlsSxlLzFdWvPSDUOI7dbG6Vji5dwp99rOtbK1PE9vYDQQDnxhjIm11h76r99k7QJgAUB8fHzN5xDxK9ZaEr/fxdyl6RSVVXDX+X2ZOrKHBn2JR7hT7jlARLX74cDuWtassdaWAduMMRlUlv1aj6QU8TE5Bwu5571UVm/OJb57O+ZNHkivsFZOxxIf4k65rwV6G2OigV3AFKDmmTCLgSuBl40xoVQepsnyZFARX+ByWV5bs4NHl28C4P5LBnDtsO400aAv8bA6y91aW26MuRNYQeXx9BettWnGmAeAddbapKrHzjPGpAMVwF3W2vz6DC7ibbbmHmX6wmTW7TjIyD4deXhiLOHtNOhL6oex1plD3/Hx8XbdunWOvLZIQyqrcLFgdRZPfbqFFoEBzL4ohsmDu2l0gJwUY8x6a218Xev0qQiRepS6q4BpC5NJ33OY8XGdmXPJAMJCNOhL6p/KXaQeFJdV8NSnW1iwOot2LZvx3DWDGRfbxelY4kdU7iIetnb7AaYvTCYr7xiXnxHOrAtjaNMy0OlY4mdU7iIecrSknMeWb+LVb3YQ3q4Fr908lN/07uh0LPFTKncRD1i1OZd7FqWwu6CIG86M4q7z+xKsQV/iIP30iZyCQ4WlPLA0nUXf76Jnx2AW3j6cM7pr0Jc4T+UuchKstXyYupd7l6RyqLCMO8/uxZ3n9NKgL2k0VO4iJ2j/4WJmL0llRdo+Yru15pWbhjKgqwZ9SeOichdxk7WWd9fn8ODSdErKXcy4oB+3nBVNUw36kkZI5S7ihuwDhdy9KIUvM/MYGtWeeZPj6NFRg76k8VK5ixxHhcvy6jfbeWx5Bk0MzL00lquHRmrQlzR6KneRX5G5/wjTFibz/c5DjO7bkYcmxtGtbQunY4m4ReUuUkNZhYvnPt/K/32WScvmATx5xSAuPU2DvsS7qNxFqknJKeCuhRvYtPcIFw3swpxLBhDaqrnTsUROmMpdhMpBX09+spnnV2cR2qo5C649g/MGdHY6lshJU7mL3/s2K58Zi1LYlneMKUMiuHt8f9q00KAv8W4qd/FbR4rLeHT5Jv69ZicR7Vvw+i0JjOgV6nQsEY9QuYtfWrlpP/e8l8Lew8XcfFY0fz6vDy2b6a+D+A79NItfOXCslAfeT2Pxj7vpHdaKxDvOZHBkO6djiXicyl38grWWpcl7mJOURkFRGb8f05v/ObsnzZtq0Jf4JpW7+Lx9h4uZ+V4qn2zcx8DwNrx+awL9Ord2OpZIvVK5i8+y1vL22mweWraR0nIXM8f358YRURr0JX5B5S4+aWd+ITMWJfP11nwSotvz6OSBRIUGOx1LpMGo3MWnVLgsL321jb9+lEHTJk14eGIcU4ZEaNCX+B2Vu/iMjL1HmJaYzIbsQ5zTL4yHJsbSpY0GfYl/UrmL1ystd/HM55nMX5lJSFAgT005jUsGddWgL/FrKnfxahuyDzFtYTIZ+44w4bSu3HtRDB006EtE5S7eqai0gr99nMG/vtxGWEgQL1wXz7kxnZyOJdJoqNzF63y9NY+7F6WwI7+QqxIimXFBP1oHadCXSHUqd/Eah4vLeGTZJt78bifdO7TkjVsTOLOnBn2J1MatT3MYY8YZYzKMMZnGmBnHWXeZMcYaY+I9F1EEPknfx9i/reLttTuZOrIHy/8wUsUuchx17rkbYwKA+cBYIAdYa4xJstam11gXAvwe+LY+gop/yj9awv3vp5O0YTf9Ooew4Np4BkW0dTqWSKPnzmGZoUCmtTYLwBjzFjABSK+xbi7wGPAXjyYUv2StJWnDbuYkpXG0pJw/ntuHO0b3pFlTjQ4QcYc75d4NyK52PwdIqL7AGHM6EGGtXWqMUbnLKdlTUMSs91L5dNN+Totoy2OXDaRPpxCnY4l4FXfKvbZPgtifHzSmCfAkcEOdT2TMVGAqQGRkpHsJxW+4XJY31+7kkWWbKHe5mHVhf24cEU2ARgeInDB3yj0HiKh2PxzYXe1+CBALfF71icDOQJIx5hJr7brqT2StXQAsAIiPj7eIVNmWd4wZicl8u+0AZ/bswLxJA4ns0NLpWCJey51yXwv0NsZEA7uAKcBVPz1orS0Afj5twRjzOfCXmsUuUpvyChcvfrWNJz7aTLOmTXh0chy/jY/Q6ACRU1RnuVtry40xdwIrgADgRWttmjHmAWCdtTapvkOKb9q45zDTE5NJzilgbEwnHrw0lk6tg5yOJeIT3PoQk7V2GbCsxtfu/ZW1o089lviykvIK5q/cyjMrM2nTIpCnrzqdC+O6aG9dxIP0CVVpUN/vPMj0hcls2X+Uiad3496LYmgX3MzpWCI+R+UuDaKwtJy/rtjMS19vo3PrIF66YQhn9wtzOpaIz1K5S737KjOPGYuSyT5QxDXDIpk+rh8hGvQlUq9U7lJvCorKePiDjby9Lpvo0GDenjqMhB4dnI4l4hdU7lIvPkrby6zFqeQfK+X2UT35f+f2JigwwOlYIn5D5S4elXukhDnvp/FB8h76d2nNv64fQlx4G6djifgdlbt4hLWW937YxQNL0yksqeAv5/XhtlE9CQzQoC8RJ6jc5ZTtOlTEzPdS+Dwjl8GRlYO+eoVp0JeIk1TuctJcLsvr3+5g3oebcFm47+IYrhsepUFfIo2Ayl1OSlbuUWYkpvDd9gP8pncoD0+MI6K9Bn2JNBYqdzkh5RUunv9iG09+spmgpk14/LKBXHZGuEYHiDQyKndxW/ruw0xL3EDqrsOcP6ATcyfEEqZBXyKNkspd6lRcVsHTn2Xy3KqttG3ZjGevHswFcV2cjiUix6Fyl+Nav+MA0xYmszX3GJMHhzP7ov60balBXyKNncpdanWspJzHV2Twyjfb6dqmBa/cNJRRfTo6HUtE3KRyl19YvTmXuxelsLugiOuGdeeucf1o1Vw/KiLeRH9j5WcFhWXM/SCdhetz6NExmHduG86QqPZOxxKRk6ByFwCWp+5h9pI0Dhwr5Xeje/L7MRr0JeLNVO5+bv+RYu5bksaHqXuJ6dKal24YQmw3DfoS8XYqdz9lrWXh+hwe/GAjRWUV3HV+X6aO7KFBXyI+QuXuh7IPFHLPeyl8sSWP+O7tmDd5IL3CWjkdS0Q8SOXuR1wuy6vfbOexFRkY4IEJA7gmoTtNNOhLxOeo3P1E5v6jzEhMZt2Og4zs05GHJ8YS3k6DvkR8lcrdx5VVuFiwOounPtlCi2YBPHH5ICYN7qZBXyI+TuXuw1J3FTBtYTLpew4zPq4z918SS8eQ5k7HEpEGoHL3QcVlFTz16RYWrM6ifXAznrtmMONiNehLxJ+o3H3M2u0HmL4wmay8Y/w2PpyZ42No0zLQ6Vgi0sBU7j7iaEk5jy3fxKvf7CC8XQv+fXMCZ/UOdTqWiDhE5e4DVmbsZ+aiFPYcLubGEVH85by+BGvQl4hfUwN4sYPHSpm7NJ1FP+yiV1grFt5+Jmd0b+d0LBFpBFTuXshay7KUvdyXlMqhwjL+95xe3HlOL5o31aAvEankVrkbY8YBTwEBwAvW2nk1Hv8TcAtQDuQCN1lrd3g4qwD7Dxcza3EqH6XvI65bG169KYGYrq2djiUijUyd5W6MCQDmA2OBHGCtMSbJWptebdkPQLy1ttAYcwfwGHBFfQT2V9Za3l2Xw9wP0iktd3H3Bf24+axommrQl4jUwp0996FAprU2C8AY8xYwAfi53K21K6utXwNc48mQ/i77QCF3L0rhy8w8hka3Z96kOHp01KAvEfl17pR7NyC72v0cIOE4628GPqztAWPMVGAqQGRkpJsR/VeFy/LK19t5fEUGAU0MD14ay1VDIzXoS0Tq5E6519YkttaFxlwDxAOjanvcWrsAWAAQHx9f63NIpS37jjAtMZkfdh5idN+OPDwxjq5tWzgdS0S8hDvlngNEVLsfDuyuucgYcy4wExhlrS3xTDz/U1ru4rlVW3n6s0yCmwfw9ytOY8JpXTXoS0ROiDvlvhbobYyJBnYBU4Crqi8wxpwO/BMYZ63d7/GUfiI55xDTFiazae8RLh7UlfsujiG0lQZ9iciJq7PcrbXlxpg7gRVUngr5orU2zRjzALDOWpsEPA60At6t2sPcaa29pB5z+5Tisgqe/Hgzz3+RRceQ5jx/XTxjYzo5HUtEvJhb57lba5cBy2p87d5qt8/1cC6/sSYrnxmJyWzPL+TKoRHMuKA/bVpo0JeInBp9QtUhR4rLmPfhJl7/dieR7Vvyxi0JnNlLg75ExDNU7g74bNM+Zr6Xyr7DxdxyVjR/Oq8PLZvpWyEinqNGaUAHjpXywPtpLP5xN73DWvHMHWdyeqQGfYmI56ncG4C1lveT9zAnKY0jxWX8YUxvfnd2Tw36EpF6o3KvZ3sLKgd9fbJxH4PC2/DoZQn066xBXyJSv1Tu9cRay1trs3n4g42UuVzMHN+fm86KJkCjA0SkAajc68GO/GPMSEzhm6x8hvVoz7xJA4kKDXY6loj4EZW7B1W4LC99tY2/fpRBYJMmPDwxjilDIjToS0QanMrdQzL2Vg762pB9iDH9wnhwYixd2mjQl4g4Q+V+ikrLXTzzeSbzV2YSEhTIP648nYsHdtGgLxFxlMr9FPyYfYjpC5PJ2HeECad15b6LB9A+uJnTsUREVO4no6i0gic+yuDFr7YRFhLEv66PZ0x/DfoSkcZD5X6Cvt6ax4zEFHYeKOSqhEhmXNCP1kEa9CUijYvK3U2Hi8t4ZNlG3vwum+4dWvLmrcMY3rOD07FERGqlcnfDJ+n7mLk4hdwjJUwd2YM/ntuHFs00OkBEGi+V+3HkHy1hzvvpvL9hN/06h7Dg2ngGRbR1OpaISJ1U7rWw1rLkx93c/34aR0vK+dPYPtw+qifNmjZxOpqIiFtU7jXsPlTErMWpfLZpP6dFtOWxywbSp1OI07FERE6Iyr2Ky2V547udzPtwExUuy+yLYrjhzCgN+hIRr6RyB7blHWNGYjLfbjvAiF4deGTiQCI7tHQ6lojISfPrci+vcPGvL7fxt48306xpEx6dHMdv4yM0OkBEvJ7flvvGPYeZnphMck4BY2M68eClsXRqHeR0LBERj/C7ci8pr2D+Z5k88/lW2rYMZP5Vgxkf11l76yLiU/yq3NfvOMj0xGQy9x9l0undmH1RDO006EtEfJBflHthaTmPr8jg5a+306V1EC/dOISz+4Y5HUtEpN74fLl/uSWPGYuSyTlYxLXDujNtXF9CNOhLRHycz5Z7QVEZD32QzjvrcogODebtqcNI6KFBXyLiH3yy3Fek7WX24lTyj5Vyx+ie/GFMb4ICNehLRPyHT5V77pES5iSl8UHKHvp3ac2/rh9CXHgbp2OJiDQ4nyh3ay2Lvt/FA0vTKSqt4K7z+zJ1ZA8CAzToS0T8k1vlbowZBzwFBAAvWGvn1Xi8OfAqcAaQD1xhrd3u2ai123WoiHsWpbBqcy6DIysHffUK06AvEfFvdZa7MSYAmA+MBXKAtcaYJGtterVlNwMHrbW9jDFTgEeBK+oj8E9cLsu/v93Box9uwgJzLo7h2uEa9CUiAu7tuQ8FMq21WQDGmLeACUD1cp8AzKm6vRB42hhjrLXWg1l/tjX3KDMSk1m7/SC/6R3KwxPjiGivQV8iIj9xp9y7AdnV7ucACb+2xlpbbowpADoAeZ4IWd07a7OZtSSVoKZNePyygVx2RrhGB4iI1OBOudfWnDX3yN1ZgzFmKjAVIDIy0o2X/qXojsGM6RfG/RMGEBaiQV8iIrVxp9xzgIhq98OB3b+yJscY0xRoAxyo+UTW2gXAAoD4+PiTOmQzJKo9Q6Lan8xvFRHxG+6cK7gW6G2MiTbGNAOmAEk11iQB11fdvgz4rL6Ot4uISN3q3HOvOoZ+J7CCylMhX7TWphljHgDWWWuTgH8BrxljMqncY59Sn6FFROT43DrP3Vq7DFhW42v3VrtdDFzu2WgiInKy9BFOEREfpHIXEfFBKncRER+kchcR8UEqdxERH2ScOh3dGJML7DjJ3x5KPYw2aOS0zf5B2+wfTmWbu1trO9a1yLFyPxXGmHXeQRnCAAAEoklEQVTW2ninczQkbbN/0Db7h4bYZh2WERHxQSp3EREf5K3lvsDpAA7QNvsHbbN/qPdt9spj7iIicnzeuucuIiLH0ajL3RgzzhiTYYzJNMbMqOXx5saYt6se/9YYE9XwKT3LjW3+kzEm3RiTbIz51BjT3YmcnlTXNldbd5kxxhpjvP7MCne22Rjz26rvdZox5o2GzuhpbvxsRxpjVhpjfqj6+R7vRE5PMca8aIzZb4xJ/ZXHjTHmH1V/HsnGmMEeDWCtbZS/qBwvvBXoATQDNgAxNdb8Dniu6vYU4G2nczfANp8NtKy6fYc/bHPVuhBgNbAGiHc6dwN8n3sDPwDtqu6HOZ27AbZ5AXBH1e0YYLvTuU9xm0cCg4HUX3l8PPAhlVeyGwZ868nXb8x77j9fmNtaWwr8dGHu6iYAr1TdXgiMMd59QdU6t9lau9JaW1h1dw2VV8byZu58nwHmAo8BxQ0Zrp64s823AvOttQcBrLX7Gzijp7mzzRZoXXW7Db+84ptXsdauppYr0lUzAXjVVloDtDXGdPHU6zfmcq/twtzdfm2NtbYc+OnC3N7KnW2u7mYq/+X3ZnVuszHmdCDCWru0IYPVI3e+z32APsaYr4wxa4wx4xosXf1wZ5vnANcYY3KovH7E/zZMNMec6N/3E+LWxToc4rELc3sRt7fHGHMNEA+MqtdE9e+422yMaQI8CdzQUIEagDvf56ZUHpoZTeX/zr4wxsRaaw/Vc7b64s42Xwm8bK19whgznMqru8Vaa131H88R9dpfjXnP/UQuzM3xLsztRdzZZowx5wIzgUustSUNlK2+1LXNIUAs8LkxZjuVxyaTvPxNVXd/tpdYa8ustduADCrL3lu5s803A+8AWGu/AYKonMHiq9z6+36yGnO5++OFuevc5qpDFP+ksti9/Tgs1LHN1toCa22otTbKWhtF5fsMl1hr1zkT1yPc+dleTOWb5xhjQqk8TJPVoCk9y51t3gmMATDG9Key3HMbNGXDSgKuqzprZhhQYK3d47Fnd/od5TrebR4PbKbyXfaZVV97gMq/3FD5zX8XyAS+A3o4nbkBtvkTYB/wY9WvJKcz1/c211j7OV5+toyb32cD/A1IB1KAKU5nboBtjgG+ovJMmh+B85zOfIrb+yawByijci/9ZuB24PZq3+P5VX8eKZ7+udYnVEVEfFBjPiwjIiInSeUuIuKDVO4iIj5I5S4i4oNU7iIiPkjlLn7NGPN7Y8xGY8zrTmcR8SSdCil+zRizCbjAVn4KtK61AdbaigaIJXLKtOcufssY8xyVI2iTjDEFxpjXjDGfGWO2GGNurVozumrG+BtUftBExCtoz138WtW8mnjgTmAilbNrgqmcpZ5A5cf+PwBi3dm7F2kstOcu8h9LrLVF1to8YCWVM8gBvlOxi7dRuYv8R83/xv50/1hDBxE5VSp3kf+YYIwJMsZ0oHKO+lqH84icNJW7yH98R+Xx9TXAXGutV1/mTfyb3lAVAYwxc4Cj1tq/Op1FxBO05y4i4oO05y4i4oO05y4i4oNU7iIiPkjlLiLig1TuIiI+SOUuIuKDVO4iIj7o/wNpsjbGdMP3SwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "points = get_roc(y, y_pred)\n",
    "df = pd.DataFrame(points, columns=[\"tpr\", \"fpr\"])\n",
    "print(\"AUC is %.3f.\" % get_auc(y, y_pred))\n",
    "df.plot(x=\"fpr\", y=\"tpr\", label=\"roc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 利用notedown包将ipython notebook文件转为markdown文件\n",
    "notedown classifier_evaluation.ipynb --to markdown --strip > classifier_evaluation.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
